---
language:
- en
- lug
tags:
- llama-3.1
- gemma-2b
- finetuned
- english-luganda
- translation
- peft
- qlora
---

# final_model_8b_16

This model is finetuned for English-Luganda bidirectional translation tasks. It's trained using QLoRA (Quantized Low-Rank Adaptation) on the original LLaMA-3.1-8B model.

## Model Details

### Base Model Information
- Base model: unsloth/Meta-Llama-3.1-8B
- Model family: LLaMA-3.1-8B
- Type: Base
- Original model size: 8B parameters

### Training Configuration
- Training method: QLoRA (4-bit quantization)
- LoRA rank (r): 16
- LoRA alpha: 16
- Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- LoRA dropout: 0
- Learning rate: 2e-5
- Batch size: 2
- Gradient accumulation steps: 4
- Max sequence length: 2048
- Weight decay: 0.01
- Training steps: 100,000
- Warmup steps: 1000
- Save interval: 10,000 steps
- Optimizer: AdamW (8-bit)
- LR scheduler: Cosine
- Mixed precision: bf16
- Gradient checkpointing: Enabled (unsloth)

### Dataset Information
- Training data: Parallel English-Luganda corpus
- Data sources:
  - SALT dataset (salt-train-v1.4)
  - Extracted parallel sentences
  - Synthetic code-mixed data
- Bidirectional translation: Trained on both English→Luganda and Luganda→English
- Total training examples: Varies by direction

### Usage
This model uses an instruction-based prompt format:
```
Below is an instruction that describes a task,
paired with an input that provides further context.
Write a response that appropriately completes the request.

### Instruction:
Translate the following text to [target_lang]

### Input:
[input text]

### Response:
[translation]
```

## Training Infrastructure
- Trained using unsloth optimization library
- Hardware: Single A100 GPU
- Quantization: 4-bit training enabled

## Limitations
- The model is specialized for English-Luganda translation
- Performance may vary based on domain and complexity of text
- Limited to the context length of 16 tokens

## Citation and Contact
If you use this model, please cite:
- Original LLaMA-3.1 model by Meta AI
- QLoRA paper: Dettmers et al. (2023)
- unsloth optimization library