|
--- |
|
license: cc-by-nc-4.0 |
|
base_model: |
|
- Qwen/Qwen2-1.5B-Instruct |
|
tags: |
|
- data processing |
|
- slm |
|
--- |
|
|
|
<div style="width: auto; margin-left: auto; margin-right: auto; background-color:black"> |
|
<img src="https://assets-global.website-files.com/6423879a8f63c1bb18d74bfa/648818d56d04c3bdf36d71ab_Refuel_rev8-01_ts-p-1600.png" alt="Refuel.ai" style="width: 100%; min-width: 400px; display: block; margin: auto;"> |
|
</div> |
|
|
|
|
|
## Model Details |
|
|
|
RefuelLLM-2-mini, aka Qwen-2-Refueled, is a Qwen-2-1.5B base model instruction tuned on a corpus of 2750+ datasets, spanning tasks such as classification, reading comprehension, structured attribute extraction and entity resolution. We're excited to open-source the model for the community to build on top of. |
|
|
|
More details about [RefuelLLM-2-mini](https://www.refuel.ai/blog-posts/refuel-llm-2-mini), and the [RefuelLLM-2 family of models](https://www.refuel.ai/blog-posts/announcing-refuel-llm-2). |
|
|
|
**Model developers** - Refuel AI |
|
|
|
**Input** - Text only. |
|
|
|
**Output** - Text only. |
|
|
|
**Architecture** - Qwen-2-Refueled is built on top of a Qwen-2-1.5B base model. |
|
|
|
**Release Date** - Jan 8, 2025. |
|
|
|
**License** - [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/deed.en) |
|
|
|
## How to use |
|
|
|
This repository contains weights for Qwen-2-Refueled that are compatible for use with HuggingFace. See the snippet below for usage with Transformers: |
|
|
|
```python |
|
>>> import torch |
|
>>> from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
>>> model_id = "refuelai/Qwen-2-Refueled" |
|
>>> tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
>>> model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto") |
|
|
|
>>> messages = [{"role": "user", "content": "Is this comment toxic or non-toxic: RefuelLLM is the new way to label text data!"}] |
|
|
|
>>> inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to("cuda") |
|
|
|
>>> outputs = model.generate(inputs, max_new_tokens=20) |
|
>>> print(tokenizer.decode(outputs[0])) |
|
``` |
|
|
|
## Benchmarks |
|
|
|
In this section, we report the output quality results on our benchmark of labeling tasks. For details on the methodology see [here](https://www.refuel.ai/blog-posts/refuel-llm-2-mini). |
|
|
|
|
|
| Model | Size | Overall | Classification | Reading Comprehension | Structure Extraction | Entity Matching | |
|
|---------------------|-------|-----------|----------------|-----------------------|-----------------------|-----------------| |
|
| RefuelLLM-2-mini | 1.5B | **75.02%**| **72.18%** | **78.18%** | 75.18% | 80.75% | |
|
| Qwen-2-3B | 3B | 67.62% | 70.91% | 71.53% | **75.72%** | 80.75% | |
|
| Phi-3.5-mini-instruct | 3.8B | 65.63% | 70.57% | 71.89% | 65.34% | **83.53%** | |
|
| Gemma-2-2B | 2B | 64.52% | 67.99% | 67.94% | 76.01% | 39.50% | |
|
| Llama-3-3B | 3B | 55.80% | 55.81% | 65.12% | 61.50% | 55.01% | |
|
| Qwen-2-1.5B | 1.5B | 51.22% | 47.36% | 67.15% | 56.17% | 45.25% | |
|
| Llama-3-1B | 1B | 39.92% | 44.58% | 29.67% | 39.50% | 62.94% | |
|
|
|
|
|
|
|
## Limitations |
|
|
|
The Qwen-2-Refueled does not have any moderation mechanisms. We're looking forward to engaging with the community |
|
on ways to make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs. |
