English
Eluwa-1.3b / README.md
Yudhanjaya's picture
Update README.md
1fb2079
|
raw
history blame
4.23 kB
metadata
license: cc-by-nc-4.0
datasets:
  - tatsu-lab/alpaca
language:
  - en

Elupatiya: A Conversational LoRA for Facebook's OPT 1.3b Architecture

logo

Eluwa is a fine-tuned Low-Rank Adapter (LoRA) model for Facebook's OPT. It is trained on the Stanford Alpaca dataset. "Elupatiya" means "baby goat", as this model is the smallest of the lot; it uses OPT-1.3b.

This repository contains the Eluwa 1.3b 2 epoch model, which represents a significant improvements in question-answering ability compared to the default OPT 1.3b model. Below are the results of Vicuna-style testing: 80 questions in various categories, with the responses rated by GPT-4.

Model OPT 1.3b base Eluwa 1.3b 1000 iter Eluwa 1.3b 2 epoch
Generic 53 58 59
Knowledge 71 69 78
Roleplay 32 44 54
Common sense 41 55 57
Fermi 21 19 17
Counterfactual 28 29 42
Coding 16 7 8
Math 3 3 3
Writing 22 11 54
Total 287 295 372

Within the bounds of our testing, it was, in fact, suprisingly good, exceeding its 2.7b cousin.

A csv of questions, answers and GPT's reviews are also included in this repo in the /TestResults/ folder, along with the base model for comparison.

Using Eluwa

I used oobabooga's text generation UI for testing, because it lets me easily regenerate outputs, modify the conversation history passed to the model, and mess with parameters.

To load Eluwa, download OPT 6.7b from Huggingface and download both the .bin and .json file from the /model folder on this Github. Follow the instructions on the text generation UI repository to figure out where the model goes and how to load a LoRA. Eluwa goes in the /loras folder.

Training and notes

Training Eluwa is a straightforward process. It is essentially Facebook's GPT-like OPT 6.7b model, loaded in 8-bit and trained using Stanford's Alapaca dataset. Use the Colab notebook here. I've written notes in there on what the functions do.

Why "Eluwa"?

Well, the whole thing was inspiration from Alpaca, which is a LoRA based on Llama. Others adopted the trend (Cabrita, Vicuna etc). Now, in Sri Lanka, we don't have llamas (at least, I've never seen any), but we do have goats. Goats are spectacular animals. In Ragama I once beheld a goat fighting a pack of stray dogs (and winning). Then it came for me. I hit it on the head with my umbrella, whereupon which it ate the umbrella and chased me the length and breadth of the entire village.

If you can't beat em, join em. "Eluwa" means goat. Goats are fearsome, versatile, and double as the essential ingredient in mutton rolls. Everything in the known universe is either a goat, or not a goat. They're not as nice as llamas or alpacas, but they'll do.

License

Facebook's OPT has its own license. Please read it here. Alpaca is licensed for research use only. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and they note that models trained using the dataset should not be used outside of research purposes.

Eluwa, therefore, is only for research and non-commercial use, under CC BY NC 4.0. Go experiment with it, but don't use it commercially. This applies to the testing dataset.