peft-starcoder-finetuned

This model is a fine-tuned version of bigcode/starcoderbase-1b on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7358

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss
1.1611	2.8470	100	0.6569
0.845	5.6940	200	0.6875
0.7272	8.5409	300	0.6951
0.6726	11.3879	400	0.7098
0.6433	14.2349	500	0.7211
0.6115	17.0819	600	0.7309
0.5989	19.9288	700	0.7325
0.5888	22.7758	800	0.7352
0.5828	25.6228	900	0.7355
0.5851	28.4698	1000	0.7358

Framework versions

• PEFT 0.13.2
• Transformers 4.46.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.3