metadata
library_name: peft
license: mit
base_model: fxmarty/really-tiny-falcon-testing
tags:
- axolotl
- generated_from_trainer
model-index:
- name: 418e6629-dcd3-424e-b3db-50c11d0d7c6b
results: []
418e6629-dcd3-424e-b3db-50c11d0d7c6b
This model is a fine-tuned version of fxmarty/really-tiny-falcon-testing on the None dataset. It achieves the following results on the evaluation set:
- Loss: 10.9416
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.000214
- train_batch_size: 4
- eval_batch_size: 4
- seed: 140
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- training_steps: 25000
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| No log | 0.0001 | 1 | 11.0842 |
| 87.9206 | 0.0563 | 500 | 10.9867 |
| 87.8458 | 0.1127 | 1000 | 10.9740 |
| 87.8012 | 0.1690 | 1500 | 10.9673 |
| 87.7685 | 0.2254 | 2000 | 10.9628 |
| 87.7475 | 0.2817 | 2500 | 10.9602 |
| 87.7401 | 0.3381 | 3000 | 10.9575 |
| 87.7205 | 0.3944 | 3500 | 10.9554 |
| 87.6986 | 0.4508 | 4000 | 10.9532 |
| 87.6961 | 0.5071 | 4500 | 10.9514 |
| 87.677 | 0.5635 | 5000 | 10.9498 |
| 87.6867 | 0.6198 | 5500 | 10.9488 |
| 87.6773 | 0.6762 | 6000 | 10.9481 |
| 87.6696 | 0.7325 | 6500 | 10.9472 |
| 87.664 | 0.7889 | 7000 | 10.9466 |
| 87.6684 | 0.8452 | 7500 | 10.9463 |
| 87.6553 | 0.9015 | 8000 | 10.9458 |
| 87.6592 | 0.9579 | 8500 | 10.9454 |
| 87.6622 | 1.0143 | 9000 | 10.9448 |
| 87.6536 | 1.0706 | 9500 | 10.9444 |
| 87.6519 | 1.1270 | 10000 | 10.9442 |
| 87.6443 | 1.1833 | 10500 | 10.9439 |
| 87.6468 | 1.2397 | 11000 | 10.9438 |
| 87.6477 | 1.2960 | 11500 | 10.9434 |
| 87.6381 | 1.3524 | 12000 | 10.9433 |
| 87.6359 | 1.4087 | 12500 | 10.9430 |
| 87.6432 | 1.4650 | 13000 | 10.9428 |
| 87.6428 | 1.5214 | 13500 | 10.9428 |
| 87.634 | 1.5777 | 14000 | 10.9426 |
| 87.6254 | 1.6341 | 14500 | 10.9425 |
| 87.6294 | 1.6904 | 15000 | 10.9424 |
| 87.6307 | 1.7468 | 15500 | 10.9422 |
| 87.6393 | 1.8031 | 16000 | 10.9422 |
| 87.6266 | 1.8595 | 16500 | 10.9421 |
| 87.6327 | 1.9158 | 17000 | 10.9419 |
| 87.6298 | 1.9722 | 17500 | 10.9419 |
| 87.6353 | 2.0285 | 18000 | 10.9420 |
| 87.6329 | 2.0849 | 18500 | 10.9418 |
| 87.6332 | 2.1412 | 19000 | 10.9418 |
| 87.6301 | 2.1976 | 19500 | 10.9417 |
| 87.6308 | 2.2539 | 20000 | 10.9417 |
| 87.6302 | 2.3103 | 20500 | 10.9417 |
| 87.6378 | 2.3666 | 21000 | 10.9416 |
| 87.6317 | 2.4230 | 21500 | 10.9416 |
| 87.6272 | 2.4793 | 22000 | 10.9416 |
| 87.6308 | 2.5357 | 22500 | 10.9416 |
| 87.6299 | 2.5920 | 23000 | 10.9416 |
| 87.6303 | 2.6484 | 23500 | 10.9416 |
| 87.6262 | 2.7047 | 24000 | 10.9415 |
| 87.6281 | 2.7610 | 24500 | 10.9415 |
| 87.6344 | 2.8174 | 25000 | 10.9416 |
Framework versions
- PEFT 0.13.2
- Transformers 4.46.0
- Pytorch 2.5.0+cu124
- Datasets 3.0.1
- Tokenizers 0.20.1