---
license: llama2
library_name: peft
tags:
- generated_from_trainer
base_model: meta-llama/Llama-2-7b-chat-hf
model-index:
- name: Llama-2-7b-chat-hf-finetune-SWE_90_10
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Llama-2-7b-chat-hf-finetune-SWE_90_10

This model is a fine-tuned version of [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 1.8121

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 3
- eval_batch_size: 3
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: reduce_lr_on_plateau
- num_epochs: 50
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch   | Step  | Validation Loss |
|:-------------:|:-------:|:-----:|:---------------:|
| 0.8511        | 0.9992  | 1231  | 0.9063          |
| 0.836         | 1.9984  | 2462  | 0.8752          |
| 0.4611        | 2.9976  | 3693  | 0.9398          |
| 0.3698        | 3.9968  | 4924  | 1.0870          |
| 0.2231        | 4.9959  | 6155  | 1.1993          |
| 0.2689        | 5.9951  | 7386  | 1.2892          |
| 0.1975        | 6.9943  | 8617  | 1.3711          |
| 0.1591        | 7.9935  | 9848  | 1.4205          |
| 0.1284        | 8.9927  | 11079 | 1.4248          |
| 0.1301        | 9.9919  | 12310 | 1.4770          |
| 0.1298        | 10.9911 | 13541 | 1.5282          |
| 0.1526        | 11.9903 | 14772 | 1.5175          |
| 0.1313        | 12.9894 | 16003 | 1.5433          |
| 0.0817        | 13.9886 | 17234 | 1.5538          |
| 0.0869        | 14.9878 | 18465 | 1.6030          |
| 0.0788        | 15.9870 | 19696 | 1.6366          |
| 0.0783        | 16.9862 | 20927 | 1.6603          |
| 0.0768        | 17.9854 | 22158 | 1.6776          |
| 0.0751        | 18.9846 | 23389 | 1.6930          |
| 0.0727        | 19.9838 | 24620 | 1.7093          |
| 0.0769        | 20.9830 | 25851 | 1.7225          |
| 0.0692        | 21.9821 | 27082 | 1.7329          |
| 0.0713        | 22.9813 | 28313 | 1.7415          |
| 0.0737        | 23.9805 | 29544 | 1.7592          |
| 0.0628        | 24.9797 | 30775 | 1.7804          |
| 0.0688        | 25.9789 | 32006 | 1.7868          |
| 0.0648        | 26.9781 | 33237 | 1.7914          |
| 0.0622        | 27.9773 | 34468 | 1.7932          |
| 0.0608        | 28.9765 | 35699 | 1.7962          |
| 0.0595        | 29.9756 | 36930 | 1.7978          |
| 0.0575        | 30.9748 | 38161 | 1.8001          |
| 0.1117        | 31.9740 | 39392 | 1.8011          |
| 0.1038        | 32.9732 | 40623 | 1.8036          |
| 0.0987        | 33.9724 | 41854 | 1.8038          |
| 0.0922        | 34.9716 | 43085 | 1.8065          |
| 0.0912        | 35.9708 | 44316 | 1.8072          |
| 0.0901        | 36.9700 | 45547 | 1.8079          |
| 0.0913        | 37.9692 | 46778 | 1.8084          |
| 0.0821        | 38.9683 | 48009 | 1.8091          |
| 0.0868        | 39.9675 | 49240 | 1.8096          |
| 0.0823        | 40.9667 | 50471 | 1.8101          |
| 0.0806        | 41.9659 | 51702 | 1.8104          |
| 0.0813        | 42.9651 | 52933 | 1.8108          |
| 0.0795        | 43.9643 | 54164 | 1.8112          |
| 0.08          | 44.9635 | 55395 | 1.8116          |
| 0.0783        | 45.9627 | 56626 | 1.8119          |
| 0.0767        | 46.9619 | 57857 | 1.8120          |
| 0.0778        | 47.9610 | 59088 | 1.8120          |
| 0.0765        | 48.9602 | 60319 | 1.8121          |
| 0.0753        | 49.9594 | 61550 | 1.8121          |


### Framework versions

- PEFT 0.10.0
- Transformers 4.40.2
- Pytorch 2.3.0+cu121
- Datasets 2.19.1
- Tokenizers 0.19.1