squad_qa_title_v5_full_recite_full_passage_meta-llama_Llama-2-7b-hf_3e-5_lora

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3501
• Accuracy: 0.8635

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Accuracy	Validation Loss
1.3765	1.0	158	0.7571	1.1334
0.9146	2.0	317	0.7872	0.8427
0.6027	3.0	475	0.8261	0.4987
0.2157	4.0	634	0.8522	0.3006
0.1491	5.0	792	0.8620	0.2357
0.1078	6.0	951	0.8645	0.2159
0.099	7.0	1109	0.8666	0.2088
0.0973	8.0	1268	0.8663	0.2123
0.0928	9.0	1426	0.8668	0.2147
0.0863	10.0	1585	0.8672	0.2138
0.0791	11.0	1743	0.8662	0.2347
0.0753	12.0	1902	0.8664	0.2411
0.0691	13.0	2060	0.8656	0.2541
0.066	14.0	2219	0.8669	0.2489
0.0623	15.0	2377	0.8659	0.2588
0.0577	16.0	2536	0.8650	0.2835
0.0559	17.0	2694	0.8652	0.3002
0.0527	18.0	2844	0.2859	0.8656
0.0518	19.0	3003	0.2897	0.8660
0.0498	20.0	3161	0.2925	0.8659
0.0492	21.0	3320	0.3170	0.8654
0.0479	22.0	3478	0.3152	0.8656
0.0481	23.0	3637	0.3220	0.8653
0.0467	24.0	3795	0.3125	0.8658
0.0463	25.0	3954	0.3322	0.8656
0.0469	26.0	4112	0.3348	0.8651
0.0453	27.0	4271	0.3273	0.8657
0.0454	28.0	4429	0.3284	0.8651
0.0455	29.0	4588	0.3430	0.8647
0.0461	30.0	4746	0.3358	0.8655
0.0452	31.0	4905	0.3390	0.8657
0.0437	32.0	5063	0.3453	0.8657
0.0442	33.0	5222	0.3508	0.8658
0.0438	34.0	5380	0.3438	0.8659
0.0443	35.0	5539	0.3416	0.8655
0.0438	36.0	5697	0.3433	0.8657
0.0442	37.0	5856	0.3478	0.8647
0.0472	38.0	6014	0.3471	0.8648
0.0461	39.0	6173	0.3345	0.8654
0.0441	40.0	6331	0.3421	0.8656
0.0433	41.0	6490	0.3575	0.8659
0.0427	42.0	6648	0.3635	0.8658
0.0429	43.0	6807	0.3560	0.8662
0.042	44.0	6965	0.3554	0.8661
0.0427	45.0	7124	0.3583	0.8658
0.0423	46.0	7282	0.3799	0.8660
0.0422	47.0	7441	0.3888	0.8659
0.0421	48.0	7599	0.3795	0.8658
0.0419	49.0	7758	0.3671	0.8662
0.0483	49.9	7900	0.3501	0.8635

Framework versions

• Transformers 4.34.0
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.14.1