|
--- |
|
license: cc-by-nc-4.0 |
|
base_model: mistralai/Mistral-7B-Instruct-v0.2 |
|
tags: |
|
- generated_from_trainer |
|
- classification |
|
- Transformer-heads |
|
- finetune |
|
- chatml |
|
- gpt4 |
|
- synthetic data |
|
- distillation |
|
model-index: |
|
- name: Mistral_classification_head_qlora |
|
results: [] |
|
datasets: |
|
- dair-ai/emotion |
|
language: |
|
- en |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
--- |
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# Mistral_classification_head_qlora |
|
|
|
 |
|
|
|
Mistral_classification_head_qlora has a new transformer head attached to it for sequence classification task and then resulting model has been finetuned on [dair-ai/emotion](https://huggingface.co/datasets/dair-ai/emotion) |
|
dataset using QloRA. The model has been trained for 1 epoch on 1x A40 GPU. The evaluation loss for the **emotion-head-3** attached to it was **1.313**. The base model used was |
|
|
|
* **[mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)** |
|
|
|
This experiment was performed using **[Transformer-heads library](https://github.com/center-for-humans-and-machines/transformer-heads/tree/main)** |
|
|
|
### Training Script |
|
|
|
The training script for attaching a new transformer head for classification task using QLoRA is following: |
|
|
|
[Training Script Colab](https://colab.research.google.com/drive/1rPaG-Q6d_CutPOlKzjsfmPvwebNg_X6i?usp=sharing) |
|
|
|
|
|
### Evaluating the Emotion-Head-3 |
|
|
|
For evaluating the transformer head that has been attached to the base model, you can refer to the following colab notebook |
|
[Colab Notebook for Evaluation](https://colab.research.google.com/drive/15UpNnoKJIWjG3G_WJFOQebjpUWyNoPKT?usp=sharing) |
|
|
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
|
|
train_epochs = 1 |
|
eval_epochs = 1 |
|
logging_steps = 1 |
|
train_batch_size = 4 |
|
eval_batch_size = 4 |
|
|
|
* output_dir="emotion_linear_probe", |
|
* learning_rate=0.00002, |
|
* num_train_epochs=train_epochs, |
|
* logging_steps=logging_steps, |
|
* do_eval=False, |
|
* remove_unused_columns=False, |
|
* optim="paged_adamw_32bit", |
|
* gradient_checkpointing=True, |
|
* lr_scheduler_type="constant", |
|
* ddp_find_unused_parameters=False, |
|
* per_device_train_batch_size=train_batch_size, |
|
* per_device_eval_batch_size=eval_batch_size, |
|
* report_to=["wandb"] |
|
|
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.39.0.dev0 |
|
- Pytorch 2.1.2+cu118 |
|
- Datasets 2.17.0 |
|
- Tokenizers 0.15.0 |
|
- Transfomer-heads |
|
|
