gemma-2b-dolly-tuned

This model is a fine-tuned version of google/gemma-2b on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 2.0192

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training Hardware

This model was trained using Intel(R) Data Center GPU Max 1100

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.05
• training_steps: 1480

Training results

Training Loss	Epoch	Step	Validation Loss
2.9204	1.64	100	2.5693
2.4476	3.28	200	2.2830
2.2496	4.92	300	2.1630
2.141	6.56	400	2.0984
2.1131	8.2	500	2.0701
2.0799	9.84	600	2.0543
2.0593	11.48	700	2.0434
2.0529	13.11	800	2.0357
2.0433	14.75	900	2.0305
2.0322	16.39	1000	2.0264
2.0197	18.03	1100	2.0234
2.0204	19.67	1200	2.0214
2.0153	21.31	1300	2.0199
2.0101	22.95	1400	2.0192

Framework versions

• PEFT 0.10.0
• Transformers 4.39.3
• Pytorch 2.0.1a0+cxx11.abi
• Datasets 2.18.0
• Tokenizers 0.15.2