File size: 5,990 Bytes
e3bb9e5 f91ed5a 7ae9eda afcb5b5 ff9f19f be2c0c0 6be8dd1 be2c0c0 4bcc591 d5ba9d7 242b4e9 d5ba9d7 e4b7916 d5ba9d7 d504a50 d5ba9d7 6dcf195 d5ba9d7 7ff2d9e afcb5b5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 |
---
language:
- id
license: apache-2.0
tags:
- Indonesian
- Chat
- Instruct
base_model:
- meta-llama/Llama-3.2-3B-Instruct
datasets:
- NekoFi/alpaca-gpt4-indonesia-cleaned
pipeline_tag: text-generation
model-index:
- name: FinMatcha-3B-Instruct
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 75.48
name: strict accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=xMaulana/FinMatcha-3B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 23.19
name: normalized accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=xMaulana/FinMatcha-3B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 12.39
name: exact match
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=xMaulana/FinMatcha-3B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 2.57
name: acc_norm
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=xMaulana/FinMatcha-3B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 5.02
name: acc_norm
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=xMaulana/FinMatcha-3B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 24.24
name: accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=xMaulana/FinMatcha-3B-Instruct
name: Open LLM Leaderboard
---
# FinMatcha-3B-Instruct
FinMatcha is a powerful Indonesian-focused large language model (LLM) fine-tuned from the [Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) base model. The model has been trained to handle a variety of conversation, with a special emphasis on understanding and generating Indonesian text.
This model has been fine-tuned on a wide array of Indonesian datasets, making it adept at handling the nuances of the Indonesian language, from formal to colloquial speech. It also supports English for bilingual applications.
## Model Details
- **Finetuned from model**: [Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
- **Dataset**: [NekoFi/alpaca-gpt4-indonesia-cleaned](https://huggingface.co/datasets/NekoFi/alpaca-gpt4-indonesia-cleaned)
- **Model Size**: 3B
- **License**: [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
- **Languages**: Indonesian, English
## How to use
### Installation
To use the Finmatcha model, install the required dependencies:
```bash
pip install transformers>=4.45
```
### Usage
[Google Colab](https://colab.research.google.com/drive/14TuDacCjHDadOY9kFkRjvORgU-cEo3D8?usp=sharing)
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "xMaulana/FinMatcha-3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer("Bagaimanakah sebuah negara dapat terbentuk?", return_tensors="pt").to("cuda")
outputs = model.generate(inputs.input_ids,
max_new_tokens = 2048,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
temperature=0.7,
do_sample=True,
top_k=5,
top_p=0.9,
repetition_penalty=1.1
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## Limitations
- The model is primarily focused on the Indonesian language and may not perform as well on non-Indonesian tasks.
- As with all LLMs, cultural and contextual biases can be present.
## License
The model is licensed under the [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0).
## Contributing
We welcome contributions to enhance and improve Finmatcha. Feel free to open issues or submit pull requests for improvements.
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_xMaulana__FinMatcha-3B-Instruct)
| Metric |Value|
|-------------------|----:|
|Avg. |23.81|
|IFEval (0-Shot) |75.48|
|BBH (3-Shot) |23.19|
|MATH Lvl 5 (4-Shot)|12.39|
|GPQA (0-shot) | 2.57|
|MuSR (0-shot) | 5.02|
|MMLU-PRO (5-shot) |24.24|
|