|
--- |
|
language: |
|
- ku |
|
- en |
|
license: apache-2.0 |
|
library_name: transformers |
|
tags: |
|
- text-generation-inference |
|
- transformers |
|
- unsloth |
|
- mistral |
|
- gguf |
|
base_model: |
|
- nazimali/Mistral-Nemo-Kurdish |
|
datasets: |
|
- saillab/alpaca-kurdish_kurmanji-cleaned |
|
base_model_relation: finetune |
|
model-index: |
|
- name: Mistral-Nemo-Kurdish-Instruct |
|
results: |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: IFEval (0-Shot) |
|
type: HuggingFaceH4/ifeval |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: inst_level_strict_acc and prompt_level_strict_acc |
|
value: 48.6 |
|
name: strict accuracy |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: BBH (3-Shot) |
|
type: BBH |
|
args: |
|
num_few_shot: 3 |
|
metrics: |
|
- type: acc_norm |
|
value: 26.02 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MATH Lvl 5 (4-Shot) |
|
type: hendrycks/competition_math |
|
args: |
|
num_few_shot: 4 |
|
metrics: |
|
- type: exact_match |
|
value: 0.3 |
|
name: exact match |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: GPQA (0-shot) |
|
type: Idavidrein/gpqa |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: acc_norm |
|
value: 4.59 |
|
name: acc_norm |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MuSR (0-shot) |
|
type: TAUR-Lab/MuSR |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: acc_norm |
|
value: 8.84 |
|
name: acc_norm |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MMLU-PRO (5-shot) |
|
type: TIGER-Lab/MMLU-Pro |
|
config: main |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 23.19 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct |
|
name: Open LLM Leaderboard |
|
--- |
|
|
|
<div dir="auto" align="right"> |
|
ئەمە مۆدێلێکی پارامێتری 12B یە، وردکراوە لەسەر نازیماڵی/میستراڵ-نیمۆ-کوردی بۆ یەک داتا سێتی ڕێنمایی کوردی (کرمانجی). مەبەستم ئەوە بوو کە ئەمە بە هەردوو ڕێنووسی کوردی کرمانجی لاتینی و کوردی سۆرانی عەرەبی ڕابهێنم، بەڵام کاتی ڕاهێنان زۆر لەوە زیاتر بوو کە پێشبینی دەکرا. بۆیە بڕیارمدا 1 داتا سێتی کوردی کورمانجی تەواو بەکاربهێنم بۆ دەستپێکردن. |
|
|
|
سەیری ڕێکخستنی ڕاهێنانی فرە GPU دەکات بۆیە پێویست ناکات بە درێژایی ڕۆژ چاوەڕێی ئەنجامەکان بکەیت. دەتەوێت بە هەردوو ڕێنووسی عەرەبی کرمانجی و سۆرانی ڕاهێنانی پێبکەیت. |
|
|
|
نموونەی دیمۆی بۆشاییەکان تاقی بکەرەوە. |
|
</div> |
|
|
|
|
|
This is a 12B parameter model, finetuned on `nazimali/Mistral-Nemo-Kurdish` for a single Kurdish (Kurmanji) instruction dataset. My intention was to train this with both Kurdish Kurmanji Latin script and Kurdish Sorani Arabic script, but training time was much longer than anticipated. |
|
So I decided to use 1 full Kurdish Kurmanji dataset to get started. |
|
|
|
Will look into a multi-GPU training setup so don't have to wait all day for results. Want to train it with both Kurmanji and Sorani Arabic script. |
|
|
|
Try [spaces demo](https://huggingface.co/spaces/nazimali/Mistral-Nemo-Kurdish-Instruct) example. |
|
|
|
### Example usage |
|
|
|
#### llama-cpp-python |
|
|
|
```python |
|
from llama_cpp import Llama |
|
|
|
inference_prompt = """Li jêr rêwerzek heye ku peywirek rave dike, bi têketinek ku çarçoveyek din peyda dike ve tê hev kirin. Bersivek ku daxwazê bi guncan temam dike binivîsin. |
|
### Telîmat: |
|
{} |
|
### Têketin: |
|
{} |
|
### Bersiv: |
|
""" |
|
|
|
llm = Llama.from_pretrained( |
|
repo_id="nazimali/Mistral-Nemo-Kurdish-Instruct", |
|
filename="Q4_K_M.gguf", |
|
) |
|
|
|
llm.create_chat_completion( |
|
messages = [ |
|
{ |
|
"role": "user", |
|
"content": inference_prompt.format("سڵاو ئەلیکوم، چۆنیت؟") |
|
} |
|
] |
|
) |
|
``` |
|
|
|
#### llama.cpp |
|
|
|
```shell |
|
./llama-cli \ |
|
--hf-repo "nazimali/Mistral-Nemo-Kurdish-Instruct" \ |
|
--hf-file Q4_K_M.gguf \ |
|
-p "selam alikum, tu çawa yî?" \ |
|
--conversation |
|
``` |
|
|
|
#### Transformers |
|
|
|
```python |
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig |
|
|
|
infer_prompt = """Li jêr rêwerzek heye ku peywirek rave dike, bi têketinek ku çarçoveyek din peyda dike ve tê hev kirin. Bersivek ku daxwazê bi guncan temam dike binivîsin. |
|
### Telîmat: |
|
{} |
|
### Têketin: |
|
{} |
|
### Bersiv: |
|
""" |
|
|
|
model_id = "nazimali/Mistral-Nemo-Kurdish-Instruct" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
|
bnb_config = BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_use_double_quant=True, |
|
bnb_4bit_quant_type="nf4", |
|
bnb_4bit_compute_dtype=torch.bfloat16, |
|
) |
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_id, |
|
quantization_config=bnb_config, |
|
device_map="auto", |
|
) |
|
|
|
model.eval() |
|
|
|
|
|
def call_llm(user_input, instructions=None): |
|
instructions = instructions or "tu arîkarek alîkar î" |
|
prompt = infer_prompt.format(instructions, user_input) |
|
|
|
input_ids = tokenizer( |
|
prompt, |
|
return_tensors="pt", |
|
add_special_tokens=False, |
|
return_token_type_ids=False, |
|
).to("cuda") |
|
|
|
with torch.inference_mode(): |
|
generated_ids = model.generate( |
|
**input_ids, |
|
max_new_tokens=120, |
|
do_sample=True, |
|
temperature=0.7, |
|
top_p=0.7, |
|
num_return_sequences=1, |
|
pad_token_id=tokenizer.pad_token_id, |
|
eos_token_id=tokenizer.eos_token_id, |
|
) |
|
|
|
decoded_output = tokenizer.batch_decode(generated_ids)[0] |
|
|
|
return decoded_output.replace(prompt, "").replace("</s>", "") |
|
|
|
response = call_llm("سڵاو ئەلیکوم، چۆنیت؟") |
|
print(response) |
|
``` |
|
|
|
### Training |
|
|
|
Transformers `4.44.2` |
|
1 NVIDIA A40 |
|
Duration 7h 41m 12s |
|
|
|
```json |
|
{ |
|
"total_flos": 2225817933447045000, |
|
"train/epoch": 0.9998075072184792, |
|
"train/global_step": 2597, |
|
"train/grad_norm": 1.172538161277771, |
|
"train/learning_rate": 0, |
|
"train/loss": 0.7774, |
|
"train_loss": 0.892096030377038, |
|
"train_runtime": 27479.3172, |
|
"train_samples_per_second": 1.512, |
|
"train_steps_per_second": 0.095 |
|
} |
|
``` |
|
|
|
|
|
#### Finetuning data: |
|
|
|
- `saillab/alpaca-kurdish_kurmanji-cleaned` |
|
- Dataset number of rows: 52,002 |
|
- Filtered columns `instruction, output` |
|
- Must have at least 1 character |
|
- Must be less than 10,000 characters |
|
- Number of rows used for training: 41,559 |
|
|
|
#### Finetuning instruction format: |
|
|
|
```python |
|
finetune_prompt = """Li jêr rêwerzek heye ku peywirek rave dike, bi têketinek ku çarçoveyek din peyda dike ve tê hev kirin. Bersivek ku daxwazê bi guncan temam dike binivîsin. |
|
### Telîmat: |
|
{} |
|
### Têketin: |
|
{} |
|
### Bersiv: |
|
{} |
|
""" |
|
``` |
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) |
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_nazimali__Mistral-Nemo-Kurdish-Instruct) |
|
|
|
| Metric |Value| |
|
|-------------------|----:| |
|
|Avg. |18.59| |
|
|IFEval (0-Shot) |48.60| |
|
|BBH (3-Shot) |26.02| |
|
|MATH Lvl 5 (4-Shot)| 0.30| |
|
|GPQA (0-shot) | 4.59| |
|
|MuSR (0-shot) | 8.84| |
|
|MMLU-PRO (5-shot) |23.19| |
|
|
|
|
