squad_qa_title_v5_full_add3_meta-llama_Llama-2-7b-hf_3e-5_lora

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.2280
• Accuracy: 0.6805

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.4938	1.0	158	1.2449	0.6971
1.1222	2.0	317	1.2146	0.6904
0.9588	3.0	476	1.1494	0.7081
0.6516	4.0	635	1.1579	0.7004
0.5382	5.0	793	1.1923	0.7003
0.3783	6.0	952	1.2099	0.7042
0.2976	7.0	1111	1.2805	0.7016
0.2588	8.0	1270	1.3570	0.6936
0.221	9.0	1428	1.4097	0.6954
0.2046	10.0	1587	1.4708	0.6884
0.1747	11.0	1746	1.5390	0.6966
0.1595	12.0	1905	1.6042	0.6928
0.1484	13.0	2063	1.7245	0.6913
0.1418	14.0	2222	1.7843	0.6914
0.1352	15.0	2381	1.7591	0.6918
0.1332	16.0	2540	1.7855	0.6915
0.124	17.0	2698	1.8559	0.6845
0.1216	18.0	2857	1.7711	0.6939
0.1212	19.0	3016	1.8782	0.6892
0.1201	20.0	3175	1.8749	0.6939
0.116	21.0	3333	1.9779	0.6873
0.1178	22.0	3492	1.8992	0.6871
0.1126	23.0	3651	1.9506	0.6910
0.1157	24.0	3810	1.9331	0.6891
0.1101	25.0	3968	2.0093	0.6890
0.1091	26.0	4127	1.9804	0.6802
0.1059	27.0	4286	2.0450	0.6907
0.1089	28.0	4445	1.9597	0.6895
0.1106	29.0	4603	2.0416	0.6848
0.1072	30.0	4762	2.0787	0.6833
0.1092	31.0	4921	1.9773	0.6868
0.1096	32.0	5080	2.0183	0.6881
0.1077	33.0	5238	2.1192	0.6885
0.1065	34.0	5397	2.0604	0.6801
0.107	35.0	5556	2.0040	0.6794
0.1063	36.0	5715	2.0488	0.6826
0.1088	37.0	5873	2.1437	0.6851
0.1041	38.0	6032	2.1194	0.6847
0.1053	39.0	6191	2.1010	0.6840
0.1062	40.0	6350	2.1417	0.6797
0.1052	41.0	6508	2.1034	0.6847
0.1061	42.0	6667	2.0156	0.6833
0.1043	43.0	6826	2.1744	0.6825
0.1039	44.0	6985	2.1533	0.6847
0.1059	45.0	7143	2.1866	0.6836
0.1062	46.0	7302	2.1303	0.6800
0.1023	47.0	7461	2.1316	0.6850
0.1056	48.0	7620	2.1523	0.6809
0.1035	49.0	7778	2.2397	0.6815
0.1029	49.76	7900	2.2280	0.6805

Framework versions

• Transformers 4.34.0
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.14.1