--- base_model: TheBloke/Mistral-7B-Instruct-v0.2-GPTQ library_name: peft license: apache-2.0 tags: - generated_from_trainer model-index: - name: shawgpt-ft results: [] --- # shawgpt-ft This model is a fine-tuned version of [TheBloke/Mistral-7B-Instruct-v0.2-GPTQ](https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GPTQ) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 1.3086 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0003 - train_batch_size: 4 - eval_batch_size: 4 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 16 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 2 - num_epochs: 20 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-------:|:----:|:---------------:| | 4.562 | 0.9231 | 3 | 3.8136 | | 3.7584 | 1.8462 | 6 | 3.0439 | | 2.9884 | 2.7692 | 9 | 2.5154 | | 1.8103 | 4.0 | 13 | 1.9763 | | 1.9109 | 4.9231 | 16 | 1.6535 | | 1.543 | 5.8462 | 19 | 1.4627 | | 1.352 | 6.7692 | 22 | 1.3896 | | 0.9888 | 8.0 | 26 | 1.3458 | | 1.2644 | 8.9231 | 29 | 1.3269 | | 1.2035 | 9.8462 | 32 | 1.3169 | | 1.198 | 10.7692 | 35 | 1.3156 | | 0.8476 | 12.0 | 39 | 1.3114 | | 1.1265 | 12.9231 | 42 | 1.3086 | | 1.0791 | 13.8462 | 45 | 1.3138 | | 1.0711 | 14.7692 | 48 | 1.3207 | | 0.7968 | 16.0 | 52 | 1.3141 | | 1.0279 | 16.9231 | 55 | 1.3216 | | 1.0094 | 17.8462 | 58 | 1.3262 | | 0.7129 | 18.4615 | 60 | 1.3255 | ### Framework versions - PEFT 0.13.2 - Transformers 4.44.2 - Pytorch 2.4.1+cu121 - Datasets 3.0.1 - Tokenizers 0.19.1