|
|
--- |
|
|
library_name: peft |
|
|
base_model: unsloth/mistral-7b-bnb-4bit |
|
|
datasets: |
|
|
- yahma/alpaca-cleaned |
|
|
--- |
|
|
|
|
|
# Model Card for Model ID |
|
|
|
|
|
A 4bit Mistral 7B model finetuned using unsloth on T4 GPU |
|
|
|
|
|
|
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- **Finetuned from model:** unsloth/mistral-7b-bnb-4bit |
|
|
|
|
|
|
|
|
- **Repository:** https://github.com/unslothai/unsloth |
|
|
|
|
|
|
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training Data |
|
|
|
|
|
https://huggingface.co/datasets/yahma/alpaca-cleaned |
|
|
|
|
|
|
|
|
### Training Procedure |
|
|
|
|
|
#### Preprocessing |
|
|
|
|
|
Alpaca prompt template is used: |
|
|
|
|
|
``` |
|
|
alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. |
|
|
|
|
|
### Instruction: |
|
|
{} |
|
|
|
|
|
### Input: |
|
|
{} |
|
|
|
|
|
### Response: |
|
|
{}""" |
|
|
|
|
|
``` |
|
|
|
|
|
|
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
|
|
``` |
|
|
per_device_train_batch_size = 2, |
|
|
gradient_accumulation_steps = 4, |
|
|
warmup_steps = 5, |
|
|
max_steps = 60, |
|
|
learning_rate = 2e-4, |
|
|
fp16 = not torch.cuda.is_bf16_supported(), |
|
|
bf16 = torch.cuda.is_bf16_supported(), |
|
|
logging_steps = 1, |
|
|
optim = "adamw_8bit", |
|
|
weight_decay = 0.01, |
|
|
lr_scheduler_type = "linear", |
|
|
seed = 3407 |
|
|
|
|
|
``` |
|
|
|
|
|
|
|
|
|
|
|
- **Hardware Type:** T4 GPU |
|
|
- **Cloud Provider:** Google Colab |
|
|
|
|
|
|
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- PEFT 0.7.1 |
