---
library_name: transformers
datasets:
- wanadzhar913/fib-malay
- wanadzhar913/boolq-malay
language:
- ms
- en
base_model:
- mesolitica/malaysian-debertav2-base
---

### Model Details

This model was originally developed as part of the 1st place solution for the [AI Tinkerer's Hackathon in Kuala Lumpur](https://www.linkedin.com/posts/supa-ai_llms-techinnovation-llm-activity-7256832143694192640-INSI?utm_source=share&utm_medium=member_desktop)
for an LLM-as-a-Judge use case.

It is a finetune [mesolitica/malaysian-debertav2-base](https://huggingface.co/mesolitica/malaysian-debertav2-base).
We're using DeBERTa (Decoding-enhanced BERT with disentangled attention) for a Natural language inference (NLI) task.
In our case, NLI is the task of determining whether a "hypothesis" is true (*entailment*) or false (*contradiction*)
given a question-statement pair. DeBERTa was selected due to its
[SOTA performance in comparison to other models like BERT and RoBERTAa](https://wandb.ai/akshayuppal12/DeBERTa/reports/The-Next-Generation-of-Transformers-Leaving-BERT-Behind-With-DeBERTa--VmlldzoyNDM2NTk2#:~:text=What%20we%20do%20see%3A%20for,accuracy%20for%20the%20validation%20set.).

### Training Details

Overall, solely using the [Boolq-Malay](https://huggingface.co/datasets/wanadzhar913/boolq-malay) dataset (comprised of both Malay and English versions of the original [Boolq](https://huggingface.co/datasets/google/boolq) dataset),
we obtain the follwing results:

- **No. of Epochs:** 10
- **Accuracy:** 66%  
- **F1-Score:** 65%  
- **Recall:** 65%  
- **Precision:** 66%  

In the future, we can do the following to garner better results:
- Increase the `gradient_accumulation_steps` to deal with the small GPU constraints or increase the `batch_size` if we've access to a larger GPU. The reasoning is mainly to avoid [Out of Memory Errors (OOM)](https://discuss.huggingface.co/t/batch-size-vs-gradient-accumulation/5260).
- Given more compute resources, we can also increase our `patience` variable and train for more than 10 epochs.

The training notebook can be found here: https://github.com/wanadzhar913/aitinkerers-hackathon-supa-team-werecooked/blob/master/notebooks-finetuning-models/02_finetune_v1_malaysian_debertav2_base.ipynb

### Usage

```python
from transformers import AutoTokenizer, AutoConfig, pipeline, \
                         DebertaV2ForSequenceClassification

config = AutoConfig.from_pretrained('wanadzhar913/malaysian-debertav2-finetune-on-boolq')
tokenizer = AutoTokenizer.from_pretrained('wanadzhar913/malaysian-debertav2-finetune-on-boolq')
model = DebertaV2ForSequenceClassification.from_pretrained('wanadzhar913/malaysian-debertav2-finetune-on-boolq', config = config)

pipe = pipeline(
    "text-classification",
    tokenizer = tokenizer,
    model=model,
    padding=True,
    device=0,
)

# https://www.astroawani.com/berita-malaysia/belanjawan-2025-gaji-minimum-ditingkatkan-kepada-rm1-700-sebulan-492383
article = """
KUALA LUMPUR: Kerajaan bersetuju untuk menaikkan kadar gaji minimum daripada RM1,500 sebulan kepada RM1,700, berkuat kuasa 1 Februari 2025.
Perdana Menteri Datuk Seri Anwar Ibrahim sewaktu membentangkan Belanjawan 2025 Malaysia MADANI di Dewan Rakyat pada Jumaat berkata,
penstrukturan ekonomi hanya dianggap berjaya apabila rakyat meraih gaji dan upah yang bermakna untuk menjalani hidup dengan lebih selesa.
"""

pipe([('Betul ke kerajaan naikkan gaji minimum?', article)])
>>> [{'label': 'entailment', 'score': 0.8098661303520203}]

pipe([('Did the government top up minimum wage?', article)])
>>> [{'label': 'entailment', 'score': 0.9928961396217346}]

pipe([('Government naikkan gaji minimum', article)])
>>> [{'label': 'entailment', 'score': 0.7880232334136963}]
```