--- base_model: TheBloke/Mistral-7B-Instruct-v0.2-GPTQ library_name: peft license: apache-2.0 tags: - generated_from_trainer model-index: - name: shawgpt-ft results: [] --- # shawgpt-ft This model is a fine-tuned version of [TheBloke/Mistral-7B-Instruct-v0.2-GPTQ](https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GPTQ) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 1.9509 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 4 - eval_batch_size: 4 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 16 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 2 - num_epochs: 12 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-------:|:----:|:---------------:| | 4.6204 | 0.9231 | 3 | 4.0975 | | 4.2876 | 1.8462 | 6 | 3.7425 | | 3.8607 | 2.7692 | 9 | 3.3912 | | 2.5952 | 4.0 | 13 | 2.9728 | | 3.1357 | 4.9231 | 16 | 2.7291 | | 2.8442 | 5.8462 | 19 | 2.5357 | | 2.6161 | 6.7692 | 22 | 2.3637 | | 1.8139 | 8.0 | 26 | 2.1825 | | 2.2632 | 8.9231 | 29 | 2.0778 | | 2.1237 | 9.8462 | 32 | 1.9958 | | 2.069 | 10.7692 | 35 | 1.9554 | | 0.421 | 11.0769 | 36 | 1.9509 | ### Framework versions - PEFT 0.13.2 - Transformers 4.44.2 - Pytorch 2.5.0+cu121 - Datasets 3.0.2 - Tokenizers 0.19.1