Eluwa: A Conversational LoRA for Facebook's OPT Architecture
Eluwa is a fine-tuned Low-Rank Adapter (LoRA) model for Facebook's OPT 1.3b, 2.7b and 6.7b architectures. It is trained on the Stanford Alpaca dataset.
This repository contains the Eluwa 1.3b 2 epoch model, which represents a significant improvements in question-answering ability compared to the default OPT 1.3b model. Below are the results of Vicuna-style testing: 80 questions in various categories, with the responses rated by GPT-4.
| Model | OPT 1.3b base | Eluwa 1.3b 1000 iter | Eluwa 1.3b 2 epoch |
|---|---|---|---|
| Generic | 53 | 58 | 59 |
| Knowledge | 71 | 69 | 78 |
| Roleplay | 32 | 44 | 54 |
| Common sense | 41 | 55 | 57 |
| Fermi | 21 | 19 | 17 |
| Counterfactual | 28 | 29 | 42 |
| Coding | 16 | 7 | 8 |
| Math | 3 | 3 | 3 |
| Writing | 22 | 11 | 54 |
| Total | 287 | 295 | 372 |
Within the bounds of our testing, it was, in fact, suprisingly good, exceeding its 2.7b cousin.
A csv of questions, answers and GPT's reviews are also included in this repo in the /TestResults/ folder of the Eluwa github repo, along with the base model for comparison.
Using Eluwa
I used oobabooga's text generation UI for testing, because it lets me easily regenerate outputs, modify the conversation history passed to the model, and mess with parameters.
To load Eluwa, download OPT 1.3b from Huggingface and download both the .bin and .json file from the /model folder on this Github. Follow the instructions on the text generation UI repository to figure out where the model goes and how to load a LoRA. Eluwa goes in the /loras folder.
Training and notes
Training Eluwa is a straightforward process. It is essentially Facebook's GPT-like OPT 6.7b model, loaded in 8-bit and trained using Stanford's Alapaca dataset. The training code is available on the Eluwa github repo and will as-is in Google Colab.
Why "Eluwa"?
Well, the whole thing was inspiration from Alpaca, which is a LoRA based on Llama. Others adopted the trend (Cabrita, Vicuna etc). Now, in Sri Lanka, we don't have llamas (at least, I've never seen any), but we do have goats. Goats are spectacular animals. In Ragama I once beheld a goat fighting a pack of stray dogs (and winning). Then it came for me. I hit it on the head with my umbrella, whereupon which it ate the umbrella and chased me the length and breadth of the entire village.
If you can't beat em, join em. "Eluwa" means goat. Goats are fearsome, versatile, and double as the essential ingredient in mutton rolls. Everything in the known universe is either a goat, or not a goat. They're not as nice as llamas or alpacas, but they'll do.
License
Facebook's OPT has its own license. Please read it here. Alpaca is licensed for research use only. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and they note that models trained using the dataset should not be used outside of research purposes.
Eluwa, therefore, is only for research and non-commercial use, under CC BY NC 4.0. Go experiment with it, but don't use it commercially. This applies to the testing dataset.