license: cc-by-nc-4.0
datasets:
- tatsu-lab/alpaca
language:
- en
Eluwa: A Conversational LoRA for Facebook's OPT 2.7b Architecture
Eluwa is a fine-tuned Low-Rank Adapter (LoRA) model for Facebook's OPT 2.7b. It is trained on the Stanford Alpaca dataset. The idea was that OPT 2.7 was too curt (and frankly, a bit of an asshole) for a model of its size, and that we could finetune it like Alpaca did to Llama.
This repository contains the Eluwa 2.7b 2 epoch model, which represents a significant improvements in question-answering ability compared to the default OPT 2.7b model. Despite Eluwa being not too different from OPT 2.7b, it often does a lot better than default OPT2.7b at accuracy and coherency.
For example, when asked "What is the capital of Sri Lanka", OPT 2.7b generates nonsense:
A: Sri Lanka is the country where the sun rises every morning on a beautiful beach. It has many beaches, but only one city - Colombo. (The sun sets at night.)
While Eluwa correctly says,
A: Colombo.
Likewise, when asked how to become a data scientist, Eluwa tries to be useful, whereas OPT 2.7B ends up insulting the user. Below are the results of Vicuna-style testing: 80 questions in various categories, with the responses rated by GPT-4.
| Model | OPT 2.7b base | Eluwa 2.7b 1000 iter | Eluwa 2.7b 2 epoch |
|---|---|---|---|
| Generic | 22 | 44 | 57 |
| Knowledge | 35 | 60 | 72 |
| Roleplay | 29 | 38 | 58 |
| Common sense | 20 | 48 | 50 |
| Fermi | 4 | 28 | 23 |
| Counterfactual | 5 | 24 | 23 |
| Coding | 2 | 7 | 7 |
| Math | 0 | 3 | 3 |
| Writing | 8 | 19 | 19 |
| Total | 125 | 271 | 312 |
(A sheet of questions, answers and GPT's reviews are also included in this repo).
Because of its small size, Eluwa can be used as research into conversational models with older and slower hardware. To load it in a UI like oobabooga, download the model's .bin and .json files, put them in a folder inside the /loras folder, and load it with the OPT 2.7b model.