squad_qa_title_v5_full_recite_full_passage_meta-llama_Llama-2-7b-hf_1e-4_lora

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3478
• Accuracy: 0.8669

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Accuracy	Validation Loss
1.2097	1.0	158	0.8026	0.6959
0.2421	2.0	317	0.8546	0.2840
0.1304	3.0	475	0.8658	0.2053
0.1007	4.0	634	0.8667	0.2005
0.0941	5.0	792	0.8665	0.2082
0.0838	6.0	951	0.8669	0.2144
0.0745	7.0	1109	0.8673	0.2232
0.0667	8.0	1268	0.8667	0.2343
0.0602	9.0	1426	0.8664	0.2490
0.0558	10.0	1585	0.8659	0.2618
0.0519	11.0	1743	0.8674	0.2661
0.05	12.0	1902	0.8680	0.2679
0.0475	13.0	2060	0.8664	0.2857
0.0484	14.0	2219	0.8660	0.2898
0.0466	15.0	2377	0.8664	0.2856
0.0464	16.0	2536	0.8661	0.3037
0.045	17.0	2694	0.8660	0.2976
0.0459	18.0	2853	0.8660	0.2930
0.0478	19.0	3011	0.8664	0.2994
0.0444	20.0	3170	0.8665	0.3027
0.0443	21.0	3328	0.8662	0.2945
0.0432	22.0	3487	0.8665	0.3020
0.0427	23.0	3645	0.8664	0.3122
0.0436	24.0	3804	0.8663	0.3181
0.0424	25.0	3962	0.8661	0.3300
0.0442	26.0	4121	0.8662	0.3173
0.0455	27.0	4279	0.8659	0.2914
0.0464	28.0	4438	0.8663	0.3043
0.0446	29.0	4596	0.8664	0.3201
0.0427	30.0	4755	0.8666	0.3103
0.0428	31.0	4913	0.8668	0.3120
0.0422	32.0	5072	0.8665	0.3209
0.0422	33.0	5230	0.8664	0.3256
0.0426	34.0	5389	0.8665	0.3295
0.0423	35.0	5547	0.8667	0.3375
0.0421	36.0	5706	0.8666	0.3299
0.0416	37.0	5864	0.8664	0.3438
0.0429	38.0	6023	0.8657	0.3313
0.0455	39.0	6181	0.8661	0.3100
0.0433	40.0	6340	0.8663	0.3111
0.0435	41.0	6498	0.8666	0.3134
0.042	42.0	6657	0.8667	0.3188
0.042	43.0	6815	0.8668	0.3219
0.0413	44.0	6974	0.8666	0.3348
0.0416	45.0	7110	0.3498	0.8666
0.0413	46.0	7269	0.3380	0.8666
0.0418	47.0	7427	0.3580	0.8668
0.041	48.0	7586	0.3516	0.8667
0.0411	49.0	7744	0.3468	0.8669
0.0417	49.98	7900	0.3478	0.8669

Framework versions

• Transformers 4.34.0
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.14.1