lmind_hotpot_train8000_eval7405_v1_qa_Qwen_Qwen1.5-4B_5e-5_lora2
This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_hotpot_train8000_eval7405_v1_qa dataset. It achieves the following results on the evaluation set:
- Loss: 3.9023
- Accuracy: 0.4878
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 1
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- total_eval_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 20.0
Training results
Training Loss | Epoch | Step | Validation Loss | Accuracy |
---|---|---|---|---|
2.2742 | 1.0 | 250 | 2.3355 | 0.5148 |
2.1673 | 2.0 | 500 | 2.3223 | 0.5169 |
2.0189 | 3.0 | 750 | 2.3526 | 0.5157 |
1.8684 | 4.0 | 1000 | 2.4020 | 0.5140 |
1.7028 | 5.0 | 1250 | 2.4910 | 0.5100 |
1.5685 | 6.0 | 1500 | 2.5835 | 0.5072 |
1.4161 | 7.0 | 1750 | 2.6927 | 0.5036 |
1.2904 | 8.0 | 2000 | 2.8047 | 0.5009 |
1.1397 | 9.0 | 2250 | 2.9097 | 0.4979 |
1.0366 | 10.0 | 2500 | 3.0555 | 0.4951 |
0.9114 | 11.0 | 2750 | 3.1526 | 0.4928 |
0.8301 | 12.0 | 3000 | 3.2417 | 0.4928 |
0.7283 | 13.0 | 3250 | 3.3687 | 0.4894 |
0.6717 | 14.0 | 3500 | 3.5007 | 0.4899 |
0.6055 | 15.0 | 3750 | 3.5470 | 0.4894 |
0.5667 | 16.0 | 4000 | 3.6736 | 0.4876 |
0.5271 | 17.0 | 4250 | 3.7100 | 0.4870 |
0.5081 | 18.0 | 4500 | 3.7920 | 0.4877 |
0.4758 | 19.0 | 4750 | 3.8548 | 0.4868 |
0.4682 | 20.0 | 5000 | 3.9023 | 0.4878 |
Framework versions
- PEFT 0.5.0
- Transformers 4.40.2
- Pytorch 2.3.0
- Datasets 2.19.1
- Tokenizers 0.19.1
- Downloads last month
- 0
Model tree for tyzhu/lmind_hotpot_train8000_eval7405_v1_qa_Qwen_Qwen1.5-4B_5e-5_lora2
Base model
Qwen/Qwen1.5-4BDataset used to train tyzhu/lmind_hotpot_train8000_eval7405_v1_qa_Qwen_Qwen1.5-4B_5e-5_lora2
Evaluation results
- Accuracy on tyzhu/lmind_hotpot_train8000_eval7405_v1_qaself-reported0.488