|
--- |
|
license: gpl-3.0 |
|
datasets: |
|
- yahma/alpaca-cleaned |
|
language: |
|
- en |
|
pipeline_tag: text2text-generation |
|
tags: |
|
- alpaca |
|
- llama |
|
- chat |
|
--- |
|
|
|
This repository comes with LoRA checkpoint to make LLaMA into a chatbot like language model. The checkpoint is the output of instruction following fine-tuning process with the following settings on 8xA100(40G) DGX system. |
|
- Dataset: [cleaned-up Alpaca dataset](https://github.com/gururise/AlpacaDataCleaned) up to 04/06/23 |
|
- Training script: borrowed from the official [Alpaca-LoRA](https://github.com/tloen/alpaca-lora) implementation |
|
- Training script: |
|
```shell |
|
python finetune.py \ |
|
--base_model='decapoda-research/llama-30b-hf' \ |
|
--num_epochs=10 \ |
|
--cutoff_len=512 \ |
|
--group_by_length \ |
|
--output_dir='./lora-alpaca' \ |
|
--lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \ |
|
--lora_r=16 \ |
|
--batch_size=... \ |
|
--micro_batch_size=... |
|
``` |
