ft-HuggingFaceH4-zephyr-7b-beta-qlora-v3

This model is a fine-tuned version of HuggingFaceH4/zephyr-7b-beta on the None dataset. It achieves the following results on the evaluation set:

• Loss: 3.2700

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 5
• total_train_batch_size: 80
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.566	2.5	5	1.3026
1.1359	5.0	10	1.2779
0.9114	7.5	15	1.3987
0.5972	10.0	20	1.4944
0.2936	12.5	25	1.8476
0.1104	15.0	30	2.1923
0.0311	17.5	35	2.5907
0.0168	20.0	40	2.7176
0.0132	22.5	45	2.8310
0.0115	25.0	50	2.9147
0.0106	27.5	55	3.0155
0.0103	30.0	60	3.1027
0.0101	32.5	65	3.1541
0.0099	35.0	70	3.1883
0.0097	37.5	75	3.2104
0.0095	40.0	80	3.2204
0.0094	42.5	85	3.2407
0.0095	45.0	90	3.2554
0.0093	47.5	95	3.2661
0.0094	50.0	100	3.2700

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.2+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2