README.md

---
library_name: transformers
tags:
- low resource
- trans
language:
- ban
- min
- en
- id
base_model:
- Yellow-AI-NLP/komodo-7b-base
---

# Model Card for Model ID

NusaMT-7B is a large language model fine-tuned for machine translation of low-resource Indonesian languages, with a focus on Balinese and Minangkabau. Built on LLaMA2-7B and leveraging the Komodo-7B-base model, it incorporates continued pre-training on non-English monolingual data, supervised fine-tuning, data preprocessing for cleaning parallel sentences, and synthetic data generation.


## Model Details

### Model Description


- **Developed by:** William Tan
- **Model type:** Decoder-only Large Language Model
- **Language(s) (NLP):** Balinese, Minangkabau, Indonesian, English
<!-- - **License:** [More Information Needed] -->
- **Finetuned from model:** Yellow-AI-NLP/komodo-7b-base

### Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:** https://github.com/williammtan/nusamt
- **Paper:** https://arxiv.org/abs/2410.07830
- **Demo:** https://indonesiaku.com/translate

## Uses

The model is designed for:
- Bidirectional translation between English/Indonesian and low-resource Indonesian languages (currently Balinese and Minangkabau)
- Language preservation and documentation
- Cross-cultural communication
- Educational purposes and language learning

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

- Integrated into translation applications
- Used for data augmentation in low-resource language tasks
- Adapted for other Indonesian regional languages
- Used as a foundation for developing language learning tools


### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

The model is not suitable for:

- Translation of languages outside its trained scope
- General text generation or chat functionality
- Real-time translation requiring minimal latency
- Critical applications where translation errors could cause harm

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

- Limited to specific language pairs (English/Indonesian ↔ Balinese/Minangkabau)
- Performance varies between translation directions, with better results for translations into low-resource languages
- Underperforms larger models (NLLB-3.3B) in translations into high-resource languages
- May not capture all dialectal variations or cultural nuances
- Uses significantly more parameters (7 billion) compared to traditional NMT models
- Limited by the quality and quantity of available training data

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

## How to Get Started with the Model

Use the code below to get started with the model.


## Training Details

### Training Data

NusaMT: https://huggingface.co/datasets/williamhtan/NusaMT

Total parallel sentences after cleaning:
- Balinese ↔ English: 35.6k sentences
- Balinese ↔ Indonesian: 44.9k sentences
- Minangkabau ↔ English: 16.6k sentences
- Minangkabau ↔ Indonesian: 22.4k sentences

Data sources:
- NLLB Mined corpus (ODC-BY license)
- NLLB SEED dataset (CC-BY-SA license)
- BASAbaliWiki (CC-BY-SA license)
- Bible verses from Alkitab.mobi (for non-profit scholarly use)
- NusaX dataset (CC-BY-SA license)

#### Preprocessing

- Length filtering (15-500 characters)
- Word length ratio of 2
- Removal of sentences with words >20 characters
- Deduplication
- Language identification with GlotLid V3 (threshold: 0.9)
- LASER3 similarity scoring (threshold: 1.09)
- GPT-4o mini-based data cleaning

#### Training Hyperparameters

- Training regime: bfloat16 mixed precision
- LoRA rank: 16
- Learning rate: 0.002
- Batch size: 10 per device
- Epochs: 3
- Data splits: 90% training, 5% validation, 5% testing
- Loss: Causal Language Modeling (CLM)


<!-- #### Speeds, Sizes, Times [optional] -->

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

<!-- [More Information Needed] -->

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

- FLORES-200 multilingual translation benchmark
- Internal test set (5% of parallel data)


#### Metrics

- spBLEU (SentencePiece tokenized BLEU)

### Results

Performance highlights:
- Outperforms SOTA models by up to +6.69 spBLEU in translations into Balinese
- Underperforms by up to -3.38 spBLEU in translations into higher-resource languages
- Consistently outperforms GPT-3.5, GPT-4, and GPT-4o in zero-shot translation

### Table 2: spBLEU Score Comparison of the LLaMA2-7B SFT Model with Various Enhancements

| Models                        | ban → en | en → ban | ban → id | id → ban |
|-------------------------------|----------|----------|----------|----------|
| LLaMA2-7B SFT                 | 27.63    | 13.94    | 27.90    | 13.68    |
| + Monolingual Pre-training    | 31.28    | 18.92    | 28.75    | 20.11    |
| + Mono + Backtranslation      | 33.97    | 20.27    | 29.62    | 20.67    |
| + Mono + LLM Cleaner          | 33.23    | 19.75    | 29.02    | 21.16    |
| + Mono + Cleaner + Backtrans. | **35.42**| **22.15**| **31.56**| **22.95**|

This table presents spBLEU scores for various configurations of the LLaMA2-7B model, showing the impact of monolingual pre-training, backtranslation, and LLM cleaning on translation performance across different language pairs.

### Table 3: spBLEU Scores of NusaMT-7B Compared Against SoTA Models and Large GPT Models

| Models                        | ban → en | en → ban | ban → id | id → ban | min → en | en → min | min → id | id → min |
|-------------------------------|----------|----------|----------|----------|----------|----------|----------|----------|
| GPT-3.5-turbo, zero-shot      | 27.17    | 11.63    | 28.17    | 13.14    | 28.75    | 11.07    | 31.06    | 11.05    |
| GPT-4o, zero-shot             | 27.11    | 11.45    | 27.89    | 13.08    | 28.63    | 11.00    | 31.27    | 11.00    |
| GPT-4, zero-shot              | 27.20    | 11.59    | 28.41    | 13.24    | 28.51    | 10.99    | 31.00    | 10.93    |
| NLLB-600M                     | 33.96    | 16.86    | 30.12    | 15.15    | 35.05    | 19.72    | 31.92    | 17.72    |
| NLLB-1.3B                     | 37.24    | 17.73    | 32.42    | 16.21    | 38.59    | 22.79    | 34.68    | 20.89    |
| NLLB-3.3B                     | **38.57**| 17.09    | **33.35**| 14.85    | **40.61**| **24.71**| **35.20**| 22.44    |
| NusaMT-7B (Ours)              | 35.42    | **22.15**| 31.56    | **22.95**| 37.23    | 24.32    | 34.29    | **23.27**|

This table compares the performance of NusaMT-7B with state-of-the-art models and large GPT models in terms of spBLEU scores across multiple language pairs. NusaMT-7B shows significant improvements, particularly in translations into low-resource languages.



## Environmental Impact


- **Hardware Type:** 2x NVIDIA RTX 4090
- **Hours used:** 1250
- **Cloud Provider:** Runpod.io
- **Carbon Emitted:** 210 kg CO2e


## Citation

If you find this model useful, please cite the following works

```
@misc{tan2024nusamt7bmachinetranslationlowresource,
      title={NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Models}, 
      author={William Tan and Kevin Zhu},
      year={2024},
      eprint={2410.07830},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.07830}, 
}
```