|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- umarbutler/open-australian-legal-qa |
|
tags: |
|
- law |
|
- legal |
|
- australia |
|
--- |
|
# AusLegalQA |
|
|
|
AusLegalQA is a fine-tune of [Mistral-8x7B-Instruct-0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) using PEFT techniques, trained on the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus). |
|
|
|
The model achieved an eval loss of 1.1391 on a subset of 100 prompts and answers from the original dataset. |
|
|
|
The model was trained with the following hyperparameters for 3 epochs. The step with the lowest eval loss was selected (coinciding with end of epoch 2) and the resulting qLoRA (4 bits) was merged into the base model. |
|
| Hyperparameter | Value | |
|
| --- | --- | |
|
| Sequence length | 1024 | |
|
| Epochs | 2 | |
|
| Optimiser | AdamW | |
|
| Learning rate | 1e-4 | |
|
| Learning rate scheduler | Cosine | |
|
| Batch size | 1 | |
|
| Weight decay | 0.01 | |
|
| Warmup ratio | 0.05 | |
|
| LoRA rank | 64 | |
|
| LoRA alpha | 128 | |
|
| LoRA dropout | 0.1 | |
|
| LoRA target | q_proj,v_proj | |
|
| NEFTune alpha | 5 | |
|
| Flash Attention | on | |
|
|
|
## Strengths |
|
The model is strong at summarisation and short-form answers with the key details. It is more likely to provide responses which assume the user is located in Australia. Ideal use-case is in a LLamaIndex/LangChain environment. |
|
|
|
## Limitations |
|
Just as the base model it does not have any moderation mechanisms. |
|
|
