Qwen3-8B LoRA Fine-tuning: Impact Of Data Scaling Explored

Nov 28, 2025 by Alex Johnson 59 views

Qwen3-8B LoRA Fine-Tuning: A Deep Dive into Data Scaling Experiments

In the realm of Natural Language Processing (NLP), fine-tuning large language models (LLMs) like Qwen3-8B is a crucial step in adapting them to specific tasks. This article delves into an experimental journey focused on Qwen3-8B LoRA fine-tuning, with a particular emphasis on the impact of data scaling. We explore how varying the size of the training dataset affects the model's performance, providing valuable insights for practitioners and researchers alike. Our experiments involve scaling the dataset from 162 samples to 477 and then to 958, meticulously analyzing the outcomes at each stage. This exploration aims to provide a comprehensive understanding of the relationship between data scale and model performance within the context of LoRA fine-tuning.

Experiment Overview

This experiment investigates the performance changes in Qwen3-8B model's LoRA fine-tuning based on the size of the training data. The primary goal is to analyze how the scale of the interview dataset impacts the learning effectiveness of the model. By systematically increasing the dataset size, we aim to pinpoint the optimal scale for fine-tuning. Furthermore, the experiment seeks to establish a clearer understanding of the interplay between LoRA parameters and dataset size, offering insights into efficient fine-tuning strategies. The study meticulously tracks key metrics such as training loss, training time, and the ratio between trainable parameters and tokens, providing a holistic view of the fine-tuning process. The results of this experiment are expected to offer valuable guidance on dataset scaling strategies for LoRA fine-tuning, ultimately contributing to more efficient and effective utilization of large language models.

Objectives

The main objectives of this study are threefold. First, we aim to validate the learning effect by increasing the size of the interview dataset used for fine-tuning. By incrementally expanding the dataset, we can observe the corresponding improvements in model performance and identify the point of diminishing returns. Second, the experiment seeks to determine the optimal dataset size for achieving the best balance between performance and computational cost. Understanding the ideal data scale is crucial for maximizing the efficiency of fine-tuning efforts. Finally, we intend to analyze the relationship between LoRA parameters and the size of the training data. This involves assessing how the configuration of LoRA parameters interacts with dataset size to influence model learning and generalization. Through these objectives, the experiment aims to provide practical insights into effective fine-tuning methodologies for LLMs.

Experimental Setup

Common LoRA Configuration

To ensure consistency and comparability across experiments, a common LoRA configuration was employed. This setup includes a lora_rank of 16, which determines the dimensionality of the low-rank matrices used in LoRA. A lora_alpha of 32 scales the LoRA weights, influencing the magnitude of updates to the pre-trained model. lora_dropout was set to 0.05 to mitigate overfitting during training. The lora_target modules were strategically selected as q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj, covering the key attention and feedforward layers within the Qwen3-8B architecture. This selection is crucial as it directly impacts the model's ability to learn and adapt to the new task. The configuration resulted in approximately 61.3 million trainable parameters, allowing for efficient fine-tuning without altering the entire model. Finally, the bias was set to none, indicating that bias terms were not included in the LoRA adapters, further streamlining the training process. This configuration provides a robust foundation for examining the impact of dataset scaling on model performance.

Common Training Settings

The experiments share a set of common training settings to maintain consistency and facilitate meaningful comparisons. The model used is Qwen3-8B, which comprises 36 layers and a model dimension (d_model) of 4096. The per_device_train_batch_size is set to 4, and the gradient_accumulation_steps is 8, resulting in an effective_batch_size of 32. This batch size is carefully chosen to balance memory usage and training stability. The learning_rate is set to 2e-5, a crucial parameter for controlling the speed and stability of convergence. An AdamW optimizer is employed for updating the model weights, known for its effectiveness in training deep neural networks. A cosine learning rate scheduler is used with a warmup_ratio of 0.03 to gradually increase the learning rate at the beginning of training, enhancing convergence. weight_decay is set to 0.01 to prevent overfitting by penalizing large weights. The models are trained for num_train_epochs of 3.0, striking a balance between thorough training and computational efficiency. Lastly, the cutoff_len is set to 10,000 tokens, defining the maximum sequence length for training, which is vital for managing memory usage and ensuring effective learning over long sequences. These settings collectively provide a controlled environment for assessing the impact of data scaling on LoRA fine-tuning.

Experimental Results

Experiment 1: Baseline (162 Samples)

In the baseline experiment, the Qwen3-8B model was fine-tuned using a dataset of 162 training samples and evaluated on 43 samples. This initial setup served as the foundation for subsequent comparisons. With 162 train samples and 43 eval samples, the training process involved 6 steps per epoch, leading to a total of 18 steps across 3 epochs. The training time for this baseline was approximately 23 minutes and 28 seconds, averaging about 62.56 seconds per step. This is a critical metric for understanding the computational cost of the fine-tuning process. The final train_loss recorded for this baseline was 1.0207, providing an initial benchmark for evaluating the effectiveness of fine-tuning. This experiment establishes a clear starting point, allowing for a direct comparison with experiments involving larger datasets and helping to quantify the impact of data scaling on model performance.

Experiment 2: 3x Expansion (477 Samples)

Building upon the baseline, Experiment 2 expanded the training dataset threefold, using 477 samples. This represented a significant increase of 315 samples, or 194%, over the baseline. The evaluation set was also expanded to 123 samples, an increase of 80 samples, or 186%. With the increased dataset size, the steps per epoch rose to 15, resulting in a total of 45 steps across the 3 epochs. The training time for this experiment was approximately 1 hour and 9 minutes (4,156 seconds), reflecting the increased computational demands of the larger dataset. The avg time per step also increased to 92.32 seconds, indicating a longer processing time for each training iteration. Notably, the final train_loss decreased to 0.8238, a substantial improvement compared to the baseline. This reduction in loss suggests that the increased dataset size facilitated more effective learning and a better fit of the model to the training data. This experiment underscores the potential benefits of scaling up training data for LoRA fine-tuning, providing empirical evidence of improved model performance with larger datasets.

Experiment 3: 6x Expansion (958 Samples)

Experiment 3 further scaled the training dataset to 958 samples, representing a 6-fold increase from the baseline. This expansion included 796 additional samples from the baseline, a significant 493% increase. The evaluation set was also substantially enlarged to 242 samples, a 199-sample increase, or 463%. With this dataset size, the training process involved 30 steps per epoch, totaling 90 steps over 3 epochs. At the time of writing, the training time for this experiment was still in progress, with an expected time of approximately 2-3 hours. This estimate highlights the increasing computational cost associated with larger datasets. The results from this experiment are highly anticipated, as they will provide further insights into the relationship between data scaling and model performance. Specifically, the evaluation of checkpoints at various stages (checkpoint-15, 30, 45) and benchmarks like KMMLU, KoBEST, and HAE-RAE are expected to reveal the extent of performance gains achieved through a larger dataset. The loss curve analysis will also be crucial in determining the convergence behavior and overall effectiveness of the training process. This experiment is pivotal in understanding the practical limits and potential benefits of scaling training data for LoRA fine-tuning.

Key Findings

The experimental results yielded several key findings that illuminate the impact of data scaling on the performance of Qwen3-8B LoRA fine-tuning. These findings provide valuable insights for optimizing fine-tuning strategies and understanding the relationship between data, model parameters, and training outcomes.

1. Loss Improvement

The most notable finding was the significant improvement in training loss as the dataset size increased. Specifically, the transition from Experiment 1 (162 samples) to Experiment 2 (477 samples) resulted in a reduction in train_loss from 1.0207 to 0.8238, representing a -19.3% improvement. This substantial decrease in loss indicates that expanding the dataset threefold led to more effective learning and a better fit of the model to the training data. The data reveals that a 3x increase in data resulted in an approximate 20% decrease in loss, highlighting the direct correlation between dataset size and model accuracy. This finding underscores the importance of sufficient training data in achieving optimal performance in LoRA fine-tuning. The trend suggests that larger datasets enable the model to generalize better and capture the underlying patterns in the data more effectively.

2. Training Time

While increasing the dataset size led to improved model performance, it also had a notable impact on training time. In Experiment 1, with 162 samples and 18 steps, the training took 23 minutes and 28 seconds. When the dataset was expanded in Experiment 2 to 477 samples and 45 steps, the training time increased to 1 hour and 9 minutes, which is approximately 2.5 times longer. This increase in training time is expected, as larger datasets require more computational resources and processing time. Additionally, the step time also saw an increase, transitioning from 62.56 seconds to 92.32 seconds, marking a +47.6% change. This highlights that as the dataset grows, each training iteration takes longer, further contributing to the overall training duration. These observations underscore the trade-off between model performance and computational cost, emphasizing the need for careful consideration of resource constraints when scaling training data.

3. LoRA Parameter-Data Ratio

Analyzing the LoRA parameter-data ratio provides crucial insights into the balance between model complexity and data availability. The table below illustrates the ratio of trainable parameters to training tokens for each experiment, comparing it against the theoretical optimal range for LoRA fine-tuning:

Experiment	Train Tokens*	Param-Token Ratio	LoRA Theoretical Range
Exp1	~1M	61.3:1	❌ Data insufficient (61M needed)
Exp2	~3.1M	19.8:1	⚠️ Still insufficient
Exp3	~6.2M	9.9:1	⚠️ Improving (92M recommended)

*Estimates based on an average of 6,500 tokens/sample

The table reveals that in Experiment 1, with approximately 1 million training tokens, the Param-Token Ratio was 61.3:1, indicating a significant deficiency in data. According to LoRA theory, an adequate amount of data would be around 61 million tokens to match the trainable parameters. In Experiment 2, the number of training tokens increased to approximately 3.1 million, improving the ratio to 19.8:1. However, this still falls short of the theoretical requirement. Experiment 3, with approximately 6.2 million tokens, further improved the ratio to 9.9:1, showing progress but remaining below the recommended 92 million tokens. These findings suggest that while increasing the dataset size improves the data-to-parameter ratio, there is still a need for more data to fully leverage the potential of LoRA fine-tuning. This underscores the importance of considering the theoretical data requirements when designing fine-tuning experiments to ensure optimal model performance.

Evaluation Setup Optimization

To optimize the evaluation process across the experiments, the evaluation strategy and save intervals were carefully configured. The goal was to strike a balance between frequent evaluations to monitor progress and efficient resource utilization. This section outlines the specific configurations used for each experiment, detailing the rationale behind the choices and their impact on the training workflow.

Baseline Experiment (162 Samples, 18 Steps)

For the baseline experiment, the eval_strategy was set to epoch, meaning that evaluation was conducted at the end of each epoch. The save_steps parameter was set to 6, which corresponds to the number of steps in each epoch. This configuration ensures that the model's performance is evaluated at the end of every epoch, providing a clear understanding of its learning progression. This strategy is well-suited for smaller datasets, where the time cost of each evaluation is relatively low, and frequent evaluations can provide valuable insights into the model's training dynamics. By evaluating at the end of each epoch, the baseline experiment provides a foundational understanding of the model's learning trajectory, which can be compared to subsequent experiments with larger datasets.

eval_strategy: epoch
save_steps: 6  # epoch마다

600 Samples Experiment (477 Samples, 45 Steps)

In the experiment with 477 samples, the evaluation strategy remained consistent with the baseline. The eval_strategy was set to epoch, and the save_steps parameter was set to 15, aligning with the number of steps per epoch for this dataset size. This ensures that the model's performance is assessed at the conclusion of each epoch, mirroring the approach used in the baseline experiment. Maintaining the same evaluation strategy allows for a direct comparison of performance metrics across experiments, providing a clear view of how data scaling influences learning. The consistent evaluation schedule enables a thorough assessment of the model's training progress, facilitating the identification of optimal checkpoints and training durations.

eval_strategy: epoch
save_steps: 15  # epoch마다

1200 Samples Experiment (958 Samples, 90 Steps)

For the largest dataset, the evaluation strategy was refined to provide more frequent assessments of the model's performance. The eval_strategy was set to steps, allowing for evaluation at specific intervals within each epoch. The eval_steps parameter was set to 15, which corresponds to half an epoch, resulting in six evaluations per training epoch. This more granular evaluation schedule provides a detailed view of the model's learning curve, enabling the identification of subtle changes in performance. The save_strategy was also set to steps, with save_steps set to 30, meaning that checkpoints were saved at the end of each epoch. This configuration allows for the preservation of model states at regular intervals, ensuring that the best-performing models are captured. The rationale behind this change is that with a larger dataset, more frequent evaluations can help in monitoring the training process closely and identifying the point of diminishing returns, where further training may not yield significant improvements. This refined strategy ensures that the training process is both efficient and informative, maximizing the value of the experimental results.

eval_strategy: steps
eval_steps: 15  # 반 epoch마다 (6번 평가)
save_strategy: steps
save_steps: 30  # epoch마다

Learning Content Summary

This section summarizes key concepts and calculations related to LoRA fine-tuning, providing a deeper understanding of the underlying mechanisms and practical considerations. We delve into the calculation of LoRA parameters, step counts, and token requirements, offering a comprehensive overview for practitioners and researchers.

LoRA Parameter Calculation Method

Understanding how LoRA parameters are calculated is crucial for optimizing fine-tuning strategies. LoRA introduces low-rank matrices to adapt the pre-trained model, reducing the number of trainable parameters. The calculation of these parameters involves several key components. Each LoRA adapter consists of two matrices, A and B, resulting in a parameter count of 2 × d_model × rank, where d_model is the model dimension and rank is the dimensionality of the low-rank matrices. For Qwen3-8B, the dimensions are as follows:

각 LoRA adapter = 2 × d_model × rank (A, B 행렬)
전체 trainable params = num_target_modules × 2 × d_model × rank

The number of trainable parameters is determined by the number of target modules, which are the specific layers or components being adapted. The formula for the total trainable parameters is num_target_modules × 2 × d_model × rank. In the context of Qwen3-8B, the model's architecture and the chosen LoRA configuration influence these parameters significantly. For the attention layers (Q, K, V, O), there are 4 modules, each contributing 2 × 4096 × 16 = 524,288 parameters per layer. For the MLP layers (gate, up, down), there are 3 modules, each contributing 2 × 12288 × 16 = 1,179,648 parameters per layer. This results in a per-layer parameter count of 1,703,936 parameters. Considering Qwen3-8B has 36 layers, the total trainable parameters amount to approximately 61.3 million. This detailed breakdown helps in understanding the resource requirements and potential scalability of LoRA fine-tuning.

Qwen3-8B:
- Attention (Q,K,V,O): 4 × (2 × 4096 × 16) = 524,288 params/layer
- MLP (gate,up,down): 3 × (2 × 12288 × 16) = 1,179,648 params/layer
- Per layer: 1,703,936 params
- Total (36 layers): 61.3M trainable parameters

Step Calculation Method

The calculation of training steps is essential for planning and executing fine-tuning experiments effectively. The number of steps per epoch and total steps are influenced by the batch size, gradient accumulation, and the size of the training dataset. The effective batch size is determined by the product of per_device_batch, gradient_accumulation, and num_gpus. This effective batch size is a critical factor in determining the memory requirements and training stability. The number of steps per epoch is then calculated by dividing the total number of samples by the effective batch size:

Effective Batch Size = per_device_batch × gradient_accumulation × num_gpus
Steps per epoch = num_samples ÷ effective_batch_size
Total steps = steps_per_epoch × num_epochs

For instance, in Experiment 3, with 958 samples and an effective batch size of 32, the steps per epoch are 958 ÷ 32 ≈ 30 steps/epoch. Over 3 epochs, this results in a total of 90 steps. Understanding these calculations allows for the optimization of training parameters and the efficient allocation of computational resources. For example, adjusting the batch size and gradient accumulation can help in maximizing GPU utilization and reducing training time. This systematic approach to step calculation ensures that the training process is both efficient and aligned with the experimental goals.

Example (Exp3):
- 958 samples ÷ 32 batch = 30 steps/epoch
- 30 × 3 epochs = 90 total steps

Token Requirements (Based on LoRA Research)

Token requirements play a crucial role in the effectiveness of LoRA fine-tuning. Based on existing research, the number of tokens required for optimal performance is related to the number of trainable parameters. The ratio between trainable parameters and tokens is a key factor in determining the amount of data needed for fine-tuning. Generally, a minimum ratio of 1:1 between trainable parameters and tokens is recommended, meaning that at least as many tokens as there are trainable parameters should be used for training. An optimal ratio of 1.5:1 is often suggested for better performance, ensuring that the model has sufficient data to learn from. For the Qwen3-8B model with approximately 61 million trainable parameters, this translates to a minimum requirement of 61 million tokens and an optimal requirement of 92 million tokens.

- **Minimum (1:1 ratio)**: trainable_params × 1 = 61M tokens
- **Optimal (1.5:1 ratio)**: trainable_params × 1.5 = 92M tokens

In the experiments conducted, the number of tokens used varied. In Experiment 3, approximately 6.2 million tokens were used, which is significantly lower than the recommended 92 million tokens. This indicates that there is room for improvement by increasing the dataset size. The current token count in Experiment 3 is an improvement from Experiments 1 and 2 but still falls short of the optimal range, suggesting that the model could benefit from more data. Understanding these token requirements helps in guiding decisions about dataset size and resource allocation, ensuring that the fine-tuning process is both effective and efficient. The analysis highlights the need for a balanced approach, where the amount of training data aligns with the complexity of the model and the specific task at hand.

- **Current (Exp3)**: ~6.2M tokens → 더 많은 데이터 필요

Next Steps

Following the initial experiments, several next steps are planned to further refine and optimize the LoRA fine-tuning process for the Qwen3-8B model. These steps include a comprehensive evaluation of the results, consideration of additional experiments, and identification of key optimization points. The goal is to build upon the current findings and develop a more robust and efficient fine-tuning strategy.

1. Evaluation Result Analysis

The immediate next step involves a thorough analysis of the evaluation results from the ongoing Experiment 3. Specifically, the checkpoints generated at steps 15, 30, and 45 will be evaluated to assess the model's performance at different stages of training. This evaluation will help in understanding the learning curve and identifying the point at which the model achieves optimal performance. The tasks include waiting for the [ ] checkpoint-15, 30, 45 (Exp3) evaluation to be completed. In addition to monitoring the training loss, the model's performance will be benchmarked using several standard evaluation datasets, including KMMLU, KoBEST, and HAE-RAE. These benchmarks provide a standardized way to compare the model's performance against other models and across different tasks. The task involves [ ] comparing the benchmarks KMMLU, KoBEST, and HAE-RAE. Finally, the training process's loss curve will be analyzed to gain insights into the model's convergence behavior and stability. The task is to [ ] analyze the loss curve to check for any irregularities or signs of overfitting. This comprehensive evaluation will inform decisions about further experimentation and optimization strategies.

2. Additional Experiment Considerations

Based on the initial results, several additional experiments are under consideration to further explore the optimal fine-tuning configuration. One potential experiment involves testing [ ] a larger dataset in the range of 1500-2000 samples to see if additional data leads to further performance improvements. This will help in determining the saturation point for data scaling and the practical limits of dataset size. Another area of exploration is the [ ] adjustment of the lora_rank. Experiments comparing different ranks (8, 16, 32) will be conducted to assess the impact of rank dimensionality on model performance. The lora_rank determines the dimensionality of the low-rank matrices used in LoRA, and finding the right balance is crucial for efficient fine-tuning. Furthermore, the [ ] optimization of the cutoff_len (currently 10K tokens) will be explored to optimize the sequence length for training. This parameter determines the maximum length of input sequences, and finding the optimal value can help in maximizing the utilization of computational resources and improving the model's ability to handle longer contexts. These additional experiments will provide a deeper understanding of the interplay between various fine-tuning parameters and their impact on model performance.

3. Optimization Points

Several optimization points have been identified to improve the efficiency and effectiveness of the LoRA fine-tuning process. One key area of focus is the [ ] improvement of training speed by reducing the time per step. This may involve optimizing the training code, utilizing more efficient hardware, or adjusting the training parameters. Reducing the training time allows for more experiments to be conducted within a given timeframe. Another focus area is the [ ] optimization of memory usage, particularly the effective utilization of GPU memory. Techniques such as gradient checkpointing and mixed-precision training may be explored to reduce memory consumption and enable the training of larger models or the use of larger batch sizes. Finally, the process of [ ] selecting the optimal checkpoint needs to be refined. This involves developing a more systematic approach to checkpoint evaluation, possibly through the use of automated metrics and validation datasets. Efficient checkpoint selection ensures that the best-performing model is identified and deployed, maximizing the benefits of fine-tuning. These optimization efforts will contribute to a more streamlined and effective fine-tuning workflow.

Relevant Files

For those interested in replicating or further exploring this work, the following files are relevant:

Config: examples/train_lora/qwen3_8b.yaml
Training logs:
- qwen3_8b_interview.log (162 samples)
- qwen3_8b_interview_600_data.log (477 samples)
- qwen3_8b_interview_1200_data.log (958 samples)
Checkpoints: saves/qwen3-8b/lora/sft/

References

LoRA vs Full Fine-tuning: The Importance of the Data-Parameter Ratio
Qwen3-8B Model Card
LLaMA-Factory Documentation

Notes

Loss improvement confirmed with data increase.
However, theoretical optimal data quantity for LoRA is still insufficient.
Actual downstream task performance measurement is necessary.

Conclusion

In conclusion, this article provided a comprehensive overview of our experimental journey into Qwen3-8B LoRA fine-tuning, focusing on the critical aspect of data scaling. We meticulously explored the impact of varying dataset sizes on model performance, uncovering valuable insights that can inform future fine-tuning endeavors. Our findings underscore the significance of sufficient training data in achieving optimal model performance, as evidenced by the substantial improvements in training loss observed with larger datasets. However, we also highlighted the trade-offs between model performance and computational cost, emphasizing the need for a balanced approach that considers both resource constraints and desired outcomes. The analysis of the LoRA parameter-data ratio further emphasized the importance of aligning dataset size with model complexity, ensuring that the model has ample data to learn from. As we move forward, the next steps outlined, including further evaluation, additional experiments, and optimization efforts, will pave the way for a more refined and efficient fine-tuning strategy.

Ultimately, this research contributes to the growing body of knowledge surrounding LLM fine-tuning, providing practical guidance for practitioners and researchers seeking to maximize the potential of these powerful models. By carefully considering the factors discussed in this article, including data scaling, evaluation strategies, and computational resources, users can effectively tailor Qwen3-8B and similar models to their specific tasks and achieve remarkable results. For more information on LoRA and other fine-tuning techniques, consider exploring resources like Hugging Face's documentation on LoRA.