Qwen2-7B_pct_ortho

This model is a fine-tuned version of unsloth/Qwen2-7B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.0710

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.02
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
2.0836	0.0206	8	1.9997
2.0451	0.0412	16	1.9894
2.0868	0.0618	24	2.0094
2.0404	0.0824	32	2.0230
2.0951	0.1031	40	2.0406
2.1037	0.1237	48	2.0564
2.1105	0.1443	56	2.0572
2.098	0.1649	64	2.0666
2.1234	0.1855	72	2.0810
2.1848	0.2061	80	2.0770
2.1566	0.2267	88	2.0833
2.1434	0.2473	96	2.0774
2.1722	0.2680	104	2.0898
2.0835	0.2886	112	2.1009
2.1355	0.3092	120	2.1047
2.1492	0.3298	128	2.0960
2.1524	0.3504	136	2.1070
2.1429	0.3710	144	2.1120
2.1611	0.3916	152	2.1227
2.1943	0.4122	160	2.1149
2.2268	0.4329	168	2.1105
2.135	0.4535	176	2.1087
2.1443	0.4741	184	2.1076
2.1925	0.4947	192	2.1068
2.1225	0.5153	200	2.1034
2.1679	0.5359	208	2.1078
2.2091	0.5565	216	2.1100
2.1175	0.5771	224	2.0976
2.1288	0.5977	232	2.1060
2.1234	0.6184	240	2.0916
2.1084	0.6390	248	2.0916
2.1631	0.6596	256	2.0923
2.1299	0.6802	264	2.0842
2.1939	0.7008	272	2.0919
2.071	0.7214	280	2.0830
2.181	0.7420	288	2.0801
2.1076	0.7626	296	2.0804
2.1185	0.7833	304	2.0761
2.1079	0.8039	312	2.0749
2.1499	0.8245	320	2.0783
2.1551	0.8451	328	2.0784
2.1117	0.8657	336	2.0784
2.1463	0.8863	344	2.0750
2.1167	0.9069	352	2.0696
2.1882	0.9275	360	2.0714
2.1131	0.9481	368	2.0716
2.1626	0.9688	376	2.0714
2.1141	0.9894	384	2.0710

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1