squad_qa_title_v5_full_qaonly_meta-llama_Llama-2-7b-hf_1e-4_lora

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.3786
• Accuracy: 0.6721

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
No log	0.99	74	1.4341	0.6835
1.593	2.0	149	1.3054	0.6927
1.0401	2.99	223	1.3988	0.6877
1.0401	3.99	298	1.4942	0.6863
0.7928	5.0	373	1.6119	0.6800
0.5087	5.99	447	1.7646	0.6772
0.3806	6.99	522	1.8426	0.6746
0.3806	8.0	597	1.9563	0.6741
0.3177	8.99	671	1.9976	0.6740
0.2785	10.0	746	2.0078	0.6762
0.2681	10.99	820	2.1315	0.6746
0.2681	11.99	895	2.1216	0.6745
0.2579	13.0	970	2.1533	0.6738
0.2477	13.99	1044	2.2242	0.6752
0.2467	14.99	1119	2.1754	0.6746
0.2467	16.0	1194	2.2735	0.6722
0.2445	16.99	1268	2.2275	0.6747
0.2412	18.0	1343	2.2674	0.6741
0.2418	18.99	1417	2.1967	0.6737
0.2418	19.99	1492	2.2366	0.6743
0.2399	21.0	1567	2.3867	0.6715
0.2334	21.99	1641	2.3286	0.6745
0.2371	22.99	1716	2.3928	0.6724
0.2371	24.0	1791	2.2756	0.6734
0.2352	24.99	1865	2.2664	0.6758
0.2343	26.0	1940	2.2839	0.6731
0.2394	26.99	2014	2.3062	0.6736
0.2394	27.99	2089	2.3525	0.6747
0.2343	29.0	2164	2.3717	0.6733
0.2309	29.99	2238	2.4194	0.6732
0.2313	30.99	2313	2.3053	0.6735
0.2313	32.0	2388	2.4772	0.6728
0.2279	32.99	2462	2.3837	0.6729
0.228	34.0	2537	2.4367	0.6723
0.2294	34.99	2611	2.3972	0.6716
0.2294	35.99	2686	2.4411	0.6723
0.2264	37.0	2761	2.4235	0.6727
0.2266	37.99	2835	2.4057	0.6738
0.2276	38.99	2910	2.4658	0.6728
0.2276	40.0	2985	2.4582	0.6731
0.2249	40.99	3059	2.5592	0.6717
0.2244	42.0	3134	2.5973	0.6715
0.2254	42.99	3208	2.6135	0.6731
0.2254	43.99	3283	2.6544	0.6719
0.2234	45.0	3358	2.6960	0.6712
0.2228	45.99	3432	2.6703	0.6710
0.2248	46.99	3507	2.6533	0.6720
0.2248	48.0	3582	2.6606	0.6714
0.2231	48.99	3656	2.5968	0.6708
0.2265	49.58	3700	2.3786	0.6721

Framework versions

• Transformers 4.34.0
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.14.1