--- language: - ku - en license: apache-2.0 library_name: transformers tags: - text-generation-inference - transformers - unsloth - mistral - gguf base_model: - nazimali/Mistral-Nemo-Kurdish datasets: - saillab/alpaca-kurdish_kurmanji-cleaned base_model_relation: finetune model-index: - name: Mistral-Nemo-Kurdish-Instruct results: - task: type: text-generation name: Text Generation dataset: name: IFEval (0-Shot) type: HuggingFaceH4/ifeval args: num_few_shot: 0 metrics: - type: inst_level_strict_acc and prompt_level_strict_acc value: 48.6 name: strict accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: BBH (3-Shot) type: BBH args: num_few_shot: 3 metrics: - type: acc_norm value: 26.02 name: normalized accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MATH Lvl 5 (4-Shot) type: hendrycks/competition_math args: num_few_shot: 4 metrics: - type: exact_match value: 0.3 name: exact match source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GPQA (0-shot) type: Idavidrein/gpqa args: num_few_shot: 0 metrics: - type: acc_norm value: 4.59 name: acc_norm source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MuSR (0-shot) type: TAUR-Lab/MuSR args: num_few_shot: 0 metrics: - type: acc_norm value: 8.84 name: acc_norm source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU-PRO (5-shot) type: TIGER-Lab/MMLU-Pro config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 23.19 name: accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nazimali/Mistral-Nemo-Kurdish-Instruct name: Open LLM Leaderboard ---
ئەمە مۆدێلێکی پارامێتری 12B یە، وردکراوە لەسەر نازیماڵی/میستراڵ-نیمۆ-کوردی بۆ یەک داتا سێتی ڕێنمایی کوردی (کرمانجی). مەبەستم ئەوە بوو کە ئەمە بە هەردوو ڕێنووسی کوردی کرمانجی لاتینی و کوردی سۆرانی عەرەبی ڕابهێنم، بەڵام کاتی ڕاهێنان زۆر لەوە زیاتر بوو کە پێشبینی دەکرا. بۆیە بڕیارمدا 1 داتا سێتی کوردی کورمانجی تەواو بەکاربهێنم بۆ دەستپێکردن. سەیری ڕێکخستنی ڕاهێنانی فرە GPU دەکات بۆیە پێویست ناکات بە درێژایی ڕۆژ چاوەڕێی ئەنجامەکان بکەیت. دەتەوێت بە هەردوو ڕێنووسی عەرەبی کرمانجی و سۆرانی ڕاهێنانی پێبکەیت. نموونەی دیمۆی بۆشاییەکان تاقی بکەرەوە.
This is a 12B parameter model, finetuned on `nazimali/Mistral-Nemo-Kurdish` for a single Kurdish (Kurmanji) instruction dataset. My intention was to train this with both Kurdish Kurmanji Latin script and Kurdish Sorani Arabic script, but training time was much longer than anticipated. So I decided to use 1 full Kurdish Kurmanji dataset to get started. Will look into a multi-GPU training setup so don't have to wait all day for results. Want to train it with both Kurmanji and Sorani Arabic script. Try [spaces demo](https://huggingface.co/spaces/nazimali/Mistral-Nemo-Kurdish-Instruct) example. ### Example usage #### llama-cpp-python ```python from llama_cpp import Llama inference_prompt = """Li jêr rêwerzek heye ku peywirek rave dike, bi têketinek ku çarçoveyek din peyda dike ve tê hev kirin. Bersivek ku daxwazê ​​bi guncan temam dike binivîsin. ### Telîmat: {} ### Têketin: {} ### Bersiv: """ llm = Llama.from_pretrained( repo_id="nazimali/Mistral-Nemo-Kurdish-Instruct", filename="Q4_K_M.gguf", ) llm.create_chat_completion( messages = [ { "role": "user", "content": inference_prompt.format("سڵاو ئەلیکوم، چۆنیت؟") } ] ) ``` #### llama.cpp ```shell ./llama-cli \ --hf-repo "nazimali/Mistral-Nemo-Kurdish-Instruct" \ --hf-file Q4_K_M.gguf \ -p "selam alikum, tu çawa yî?" \ --conversation ``` #### Transformers ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig infer_prompt = """Li jêr rêwerzek heye ku peywirek rave dike, bi têketinek ku çarçoveyek din peyda dike ve tê hev kirin. Bersivek ku daxwazê ​​bi guncan temam dike binivîsin. ### Telîmat: {} ### Têketin: {} ### Bersiv: """ model_id = "nazimali/Mistral-Nemo-Kurdish-Instruct" tokenizer = AutoTokenizer.from_pretrained(model_id) bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, ) model = AutoModelForCausalLM.from_pretrained( model_id, quantization_config=bnb_config, device_map="auto", ) model.eval() def call_llm(user_input, instructions=None): instructions = instructions or "tu arîkarek alîkar î" prompt = infer_prompt.format(instructions, user_input) input_ids = tokenizer( prompt, return_tensors="pt", add_special_tokens=False, return_token_type_ids=False, ).to("cuda") with torch.inference_mode(): generated_ids = model.generate( **input_ids, max_new_tokens=120, do_sample=True, temperature=0.7, top_p=0.7, num_return_sequences=1, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, ) decoded_output = tokenizer.batch_decode(generated_ids)[0] return decoded_output.replace(prompt, "").replace("", "") response = call_llm("سڵاو ئەلیکوم، چۆنیت؟") print(response) ``` ### Training Transformers `4.44.2` 1 NVIDIA A40 Duration 7h 41m 12s ```json { "total_flos": 2225817933447045000, "train/epoch": 0.9998075072184792, "train/global_step": 2597, "train/grad_norm": 1.172538161277771, "train/learning_rate": 0, "train/loss": 0.7774, "train_loss": 0.892096030377038, "train_runtime": 27479.3172, "train_samples_per_second": 1.512, "train_steps_per_second": 0.095 } ``` #### Finetuning data: - `saillab/alpaca-kurdish_kurmanji-cleaned` - Dataset number of rows: 52,002 - Filtered columns `instruction, output` - Must have at least 1 character - Must be less than 10,000 characters - Number of rows used for training: 41,559 #### Finetuning instruction format: ```python finetune_prompt = """Li jêr rêwerzek heye ku peywirek rave dike, bi têketinek ku çarçoveyek din peyda dike ve tê hev kirin. Bersivek ku daxwazê ​​bi guncan temam dike binivîsin. ### Telîmat: {} ### Têketin: {} ### Bersiv: {} """ ``` # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_nazimali__Mistral-Nemo-Kurdish-Instruct) | Metric |Value| |-------------------|----:| |Avg. |18.59| |IFEval (0-Shot) |48.60| |BBH (3-Shot) |26.02| |MATH Lvl 5 (4-Shot)| 0.30| |GPQA (0-shot) | 4.59| |MuSR (0-shot) | 8.84| |MMLU-PRO (5-shot) |23.19|