--- base_model: TheBloke/Mistral-7B-Instruct-v0.2-GPTQ library_name: peft license: apache-2.0 tags: - generated_from_trainer model-index: - name: shawgpt-ft-lr0.002-wd0.1 results: [] --- <!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. --> # shawgpt-ft-lr0.002-wd0.1 This model is a fine-tuned version of [TheBloke/Mistral-7B-Instruct-v0.2-GPTQ](https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GPTQ) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 1.7241 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.002 - train_batch_size: 4 - eval_batch_size: 4 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 16 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 2 - num_epochs: 12 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-------:|:----:|:---------------:| | 4.3981 | 0.9231 | 3 | 2.9899 | | 2.6742 | 1.8462 | 6 | 1.8402 | | 1.4936 | 2.7692 | 9 | 1.3598 | | 0.8827 | 4.0 | 13 | 1.3026 | | 1.0675 | 4.9231 | 16 | 1.3395 | | 0.935 | 5.8462 | 19 | 1.3912 | | 0.8225 | 6.7692 | 22 | 1.4097 | | 0.5561 | 8.0 | 26 | 1.5645 | | 0.6443 | 8.9231 | 29 | 1.5912 | | 0.5682 | 9.8462 | 32 | 1.6734 | | 0.5258 | 10.7692 | 35 | 1.7076 | | 0.0948 | 11.0769 | 36 | 1.7241 | ### Framework versions - PEFT 0.13.2 - Transformers 4.44.2 - Pytorch 2.5.0+cu124 - Datasets 3.0.1 - Tokenizers 0.19.1