---
title: ark-instruct-line-item
app_file: /home/tosi-n/ark/gorilla/app.py
sdk: gradio
sdk_version: 4.0.2
---
# ARK - Jack's Accounting ALM Training Framework

This is a base pipeline to build out our task specific repos layer on

> Note ReadMe currently for gorrila module which is mainly pipeline for long context window training flexibility. For the chimp module which is for default context window training, it has it's ReadMe in folder path

## Usage Requirements
To download and use the [pre-trained weights](#pre-trained-weights) you will need:
1. Hugging Face (HF) account with valid email. Note, the email used for HF must alse be used for the license agreement.
2. Accept the Meta [license and acceptable use policy](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) 


## Installation and Quick Guide
To install and run the application:
1. Clone the repository on your local machine, using git clone and pasting the url of this project.
2. Complete pre-requiste installation. Run the following code:
```
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
```
3. Training Pre-trained weights through Fine-tuning QLoRa, LoRa or full. Run the following code in bash script and update args before running for QloRa :
```
sh runner.sh
```
4. Merge amd get Trainable LoRA Weight. Run the following code in bash script and update args where necessary:
```
sh process_wt.sh
```
5. Test your model by terminal chat. Run the following code in bash script andupdate args where necessary:
```
sh stream.sh
```
6. Test your model on gradio UI. Run the following code in bash script and update args where necessary:
```
sh demo.sh
```


## Training args for Full and LoRA

### Fine-tuning
```
torchrun --nproc_per_node=8 fine-tune.py  \
        --model_name_or_path path_to/Llama-2-7b-hf \
        --bf16 True \
        --output_dir path_to_saving_checkpoints       \
        --cache_dir path_to_cache \
        --model_max_length 8192 \
        --use_flash_attn True \
        --low_rank_training False \
        --num_train_epochs 1  \
        --per_device_train_batch_size 1     \
        --per_device_eval_batch_size 2     \
        --gradient_accumulation_steps 8     \
        --evaluation_strategy "no"     \
        --save_strategy "steps"     \
        --save_steps 1000     \
        --save_total_limit 2     \
        --learning_rate 2e-5     \
        --weight_decay 0.0     \
        --warmup_steps 20     \
        --lr_scheduler_type "constant_with_warmup"     \
        --logging_steps 1     \
        --deepspeed "ds_configs/stage2.json" \
        --tf32 True \
        --max_steps 1000
```


### Supervised Fine-tuning
```
torchrun --nproc_per_node=8 supervised-fine-tune.py  \
        --model_name_or_path path_to_Llama2_chat_models \
        --bf16 True \
        --output_dir path_to_saving_checkpoints       \
        --model_max_length 32768 \
        --use_flash_attn True \
        --data_path LongAlpaca-12k.json \
        --low_rank_training True \
        --num_train_epochs 3  \
        --per_device_train_batch_size 1     \
        --per_device_eval_batch_size 2     \
        --gradient_accumulation_steps 1     \
        --evaluation_strategy "no"     \
        --save_strategy "steps"     \
        --save_steps 1000     \
        --save_total_limit 2     \
        --learning_rate 2e-5     \
        --weight_decay 0.0     \
        --warmup_steps 20     \
        --lr_scheduler_type "constant_with_warmup"     \
        --logging_steps 1     \
        --deepspeed "ds_configs/stage2.json" \
        --tf32 True
```

## Evaluation
### Perplexity Validation