File size: 1,067 Bytes
654016d 500db89 654016d 748db81 500db89 654016d 500db89 654016d 500db89 654016d 500db89 654016d 500db89 654016d 500db89 654016d 500db89 654016d 4207f1e 654016d 4207f1e 654016d 4207f1e 654016d 4207f1e 654016d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
library_name: peft
base_model: meta-llama/Llama-2-7b-hf
---
# Model Details
- Supervised fine-tuning (sft) based on [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) on [yahma/alpaca-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned)
- Trained with Deepspeed ZeRO-1 + TRL + QLoRA + Flash-Attntion 2 within 1h with 3090x4
- The LoRa adapter is uploaded
## Model and Training Details
- **Finetuned from model:** [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)
- **Dataset:** [yahma/alpaca-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned)
#### Preprocessing
- preprocessed and packed the sft dataset with trl.trainer.ConstantLengthDataset
### Results
### Compute Infrastructure
The model is trained using 4 * RTX 3090 - 24GB
## Model Card Authors
Yiyu (Michael) Ren
## Model Card Contact
Email: [email protected]
### Framework versions
- PEFT 0.8.2 |