0b6ca836-a2a2-412d-825a-f90dbab4d025

This model is a fine-tuned version of katuni4ka/tiny-random-falcon-40b on the None dataset. It achieves the following results on the evaluation set:

• Loss: 10.7366

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.000202
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: constant
• lr_scheduler_warmup_steps: 50
• training_steps: 400

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0016	1	11.1054
43.6621	0.0793	50	10.8431
43.3102	0.1587	100	10.8165
43.237	0.2380	150	10.7994
43.1916	0.3173	200	10.7873
43.1396	0.3967	250	10.7763
43.1061	0.4760	300	10.7662
43.0448	0.5553	350	10.7555
42.9742	0.6347	400	10.7366

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1