Hugging Face's logo

DandinPower
/
breeze_7b_lora_full_text

PEFT
Safetensors
Chinese
trl
sft
nycu-112-2-deeplearning-hw2
Generated from Trainer
Model card Files Community
DandinPower commited on
Commit
04defde
·
verified ·
1 Parent(s): 0f0a9e1

End of training

Browse files
Files changed (1) hide show
  1. README.md +98 -0
README.md ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - zh
4
+ license: apache-2.0
5
+ library_name: peft
6
+ tags:
7
+ - trl
8
+ - sft
9
+ - nycu-112-2-deeplearning-hw2
10
+ - generated_from_trainer
11
+ base_model: MediaTek-Research/Breeze-7B-Instruct-v1_0
12
+ datasets:
13
+ - DandinPower/ZH-Reading-Comprehension
14
+ model-index:
15
+ - name: breeze_7b_lora
16
+ results: []
17
+ ---
18
+
19
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
20
+ should probably proofread and complete it, then remove this comment. -->
21
+
22
+ # breeze_7b_lora
23
+
24
+ This model is a fine-tuned version of [MediaTek-Research/Breeze-7B-Instruct-v1_0](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v1_0) on the DandinPower/ZH-Reading-Comprehension dataset.
25
+ It achieves the following results on the evaluation set:
26
+ - Loss: 1.3504
27
+
28
+ ## Model description
29
+
30
+ More information needed
31
+
32
+ ## Intended uses & limitations
33
+
34
+ More information needed
35
+
36
+ ## Training and evaluation data
37
+
38
+ More information needed
39
+
40
+ ## Training procedure
41
+
42
+ ### Training hyperparameters
43
+
44
+ The following hyperparameters were used during training:
45
+ - learning_rate: 0.0001
46
+ - train_batch_size: 1
47
+ - eval_batch_size: 1
48
+ - seed: 42
49
+ - distributed_type: multi-GPU
50
+ - num_devices: 2
51
+ - gradient_accumulation_steps: 4
52
+ - total_train_batch_size: 8
53
+ - total_eval_batch_size: 2
54
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
55
+ - lr_scheduler_type: linear
56
+ - lr_scheduler_warmup_steps: 500
57
+ - num_epochs: 5.0
58
+
59
+ ### Training results
60
+
61
+ | Training Loss | Epoch | Step | Validation Loss |
62
+ |:-------------:|:------:|:----:|:---------------:|
63
+ | 2.6567 | 0.1845 | 250 | 2.6359 |
64
+ | 2.5304 | 0.3690 | 500 | 2.5482 |
65
+ | 2.4385 | 0.5535 | 750 | 2.4359 |
66
+ | 2.3947 | 0.7380 | 1000 | 2.3351 |
67
+ | 2.2359 | 0.9225 | 1250 | 2.2414 |
68
+ | 1.9919 | 1.1070 | 1500 | 2.1528 |
69
+ | 1.9533 | 1.2915 | 1750 | 2.0739 |
70
+ | 1.8919 | 1.4760 | 2000 | 1.9973 |
71
+ | 1.8247 | 1.6605 | 2250 | 1.9203 |
72
+ | 1.6582 | 1.8450 | 2500 | 1.8425 |
73
+ | 1.4947 | 2.0295 | 2750 | 1.7883 |
74
+ | 1.4298 | 2.2140 | 3000 | 1.7411 |
75
+ | 1.4936 | 2.3985 | 3250 | 1.6912 |
76
+ | 1.3752 | 2.5830 | 3500 | 1.6467 |
77
+ | 1.3758 | 2.7675 | 3750 | 1.5994 |
78
+ | 1.2897 | 2.9520 | 4000 | 1.5617 |
79
+ | 1.0563 | 3.1365 | 4250 | 1.5384 |
80
+ | 1.0315 | 3.3210 | 4500 | 1.5103 |
81
+ | 1.0657 | 3.5055 | 4750 | 1.4766 |
82
+ | 1.0247 | 3.6900 | 5000 | 1.4505 |
83
+ | 1.0058 | 3.8745 | 5250 | 1.4253 |
84
+ | 0.8809 | 4.0590 | 5500 | 1.4120 |
85
+ | 0.8298 | 4.2435 | 5750 | 1.3935 |
86
+ | 0.9152 | 4.4280 | 6000 | 1.3781 |
87
+ | 0.8512 | 4.6125 | 6250 | 1.3650 |
88
+ | 0.9111 | 4.7970 | 6500 | 1.3536 |
89
+ | 0.8168 | 4.9815 | 6750 | 1.3504 |
90
+
91
+
92
+ ### Framework versions
93
+
94
+ - PEFT 0.10.0
95
+ - Transformers 4.40.0
96
+ - Pytorch 2.2.2+cu121
97
+ - Datasets 2.19.0
98
+ - Tokenizers 0.19.1