pythia-70m_tatsu-lab_alpaca_farm_sftsd0_policy_pythia-6.9b_gold_offsetbias-8b_noise0.25_rmsd2

This model is a fine-tuned version of RylanSchaeffer/EleutherAI_pythia-70m_tatsu-lab_alpaca_farm_sftseed0 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.7678
• Accuracy: 0.5463

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 2
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.025
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
No log	0	0	0.8702	0.5243
0.8281	0.0648	100	0.8719	0.5301
0.8485	0.1296	200	0.8554	0.5347
0.9002	0.1944	300	0.8348	0.5278
0.8015	0.2592	400	0.8197	0.5355
0.9028	0.3239	500	0.8114	0.5363
0.8466	0.3887	600	0.8089	0.5262
0.787	0.4535	700	0.8041	0.5328
0.8061	0.5183	800	0.8012	0.5332
0.7956	0.5831	900	0.8038	0.5386
0.7835	0.6479	1000	0.7963	0.5413
0.8013	0.7127	1100	0.7914	0.5309
0.7416	0.7775	1200	0.7848	0.5448
0.7697	0.8422	1300	0.7822	0.5421
0.808	0.9070	1400	0.7830	0.5405
0.8269	0.9718	1500	0.7858	0.5390
0.7523	1.0366	1600	0.7834	0.5486
0.776	1.1014	1700	0.7846	0.5324
0.7691	1.1662	1800	0.7787	0.5367
0.7043	1.2310	1900	0.7805	0.5390
0.7831	1.2958	2000	0.7752	0.5351
0.7664	1.3605	2100	0.7766	0.5351
0.7473	1.4253	2200	0.7766	0.5432
0.7859	1.4901	2300	0.7778	0.5394
0.7264	1.5549	2400	0.7704	0.5424
0.8054	1.6197	2500	0.7742	0.5316
0.749	1.6845	2600	0.7729	0.5459
0.7695	1.7493	2700	0.7731	0.5367
0.7338	1.8141	2800	0.7750	0.5363
0.7656	1.8788	2900	0.7707	0.5494
0.7728	1.9436	3000	0.7717	0.5374
0.7468	2.0084	3100	0.7690	0.5413
0.7464	2.0732	3200	0.7720	0.5363
0.7817	2.1380	3300	0.7709	0.5463
0.7381	2.2028	3400	0.7705	0.5451
0.785	2.2676	3500	0.7738	0.5374
0.7441	2.3324	3600	0.7705	0.5424
0.7428	2.3971	3700	0.7662	0.5436
0.7218	2.4619	3800	0.7691	0.5394
0.77	2.5267	3900	0.7681	0.5436
0.7899	2.5915	4000	0.7697	0.5367
0.7452	2.6563	4100	0.7684	0.5451
0.7351	2.7211	4200	0.7646	0.5424
0.7531	2.7859	4300	0.7723	0.5467
0.7816	2.8507	4400	0.7714	0.5309
0.7572	2.9155	4500	0.7689	0.5405
0.7108	2.9802	4600	0.7685	0.5355
0.7955	3.0450	4700	0.7677	0.5424
0.7797	3.1098	4800	0.7686	0.5409
0.758	3.1746	4900	0.7719	0.5370
0.7649	3.2394	5000	0.7681	0.5448
0.7707	3.3042	5100	0.7697	0.5405
0.8002	3.3690	5200	0.7706	0.5382
0.7911	3.4338	5300	0.7664	0.5490
0.7788	3.4985	5400	0.7702	0.5432
0.7607	3.5633	5500	0.7695	0.5436
0.7738	3.6281	5600	0.7698	0.5417
0.7299	3.6929	5700	0.7671	0.5525
0.7466	3.7577	5800	0.7703	0.5332
0.7931	3.8225	5900	0.7704	0.5478
0.7406	3.8873	6000	0.7702	0.5413
0.751	3.9521	6100	0.7716	0.5436
0.8067	4.0168	6200	0.7675	0.5332
0.7451	4.0816	6300	0.7674	0.5370
0.775	4.1464	6400	0.7672	0.5486
0.7677	4.2112	6500	0.7681	0.5397
0.7535	4.2760	6600	0.7708	0.5459
0.7792	4.3408	6700	0.7714	0.5417
0.7675	4.4056	6800	0.7669	0.5490
0.7379	4.4704	6900	0.7693	0.5409
0.7922	4.5351	7000	0.7639	0.5467
0.7468	4.5999	7100	0.7657	0.5463
0.77	4.6647	7200	0.7684	0.5363
0.7371	4.7295	7300	0.7701	0.5451
0.8006	4.7943	7400	0.7706	0.5424
0.8371	4.8591	7500	0.7669	0.5467
0.8163	4.9239	7600	0.7696	0.5386
0.7682	4.9887	7700	0.7684	0.5455

Framework versions

• Transformers 4.42.4
• Pytorch 2.3.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1