README.md

---
license: llama3
base_model: catallama/CataLlama-v0.1-Base
tags:
- llama
- llama-3
- Catalan
model-index:
- name: CataLlama-v0.1-Instruct-SFT
  results: []
datasets:
- catallama/Catalan-Instruct
language:
- ca
- en
pipeline_tag: text-generation
---

# NOTE: [CataLlama-v0.2](https://huggingface.co/catallama/CataLlama-v0.2-Instruct-SFT-DPO-Merged) is out. Please use that one instead.


![](https://huggingface.co/catallama/CataLlama-v0.1-Instruct-DPO/resolve/main/CataLlama-v0.1.png)


# NOTE: [CataLlama-v0.2](https://huggingface.co/catallama/CataLlama-v0.2-Instruct-SFT-DPO-Merged) is out. Please use that one instead.



# CataLlama-v0.1-Instruct-SFT

**CataLlama-v0.1-Instruct-SFT** is an instruct fine-tune of [catallama/CataLlama-v0.1-Base](https://huggingface.co/catallama/CataLlama-v0.1-Base) on the [catallama/Catalan-Instruct](https://huggingface.co/datasets/catallama/Catalan-Instruct) dataset.

CataLlama was trained on roughly **445 million new tokens** in three separate stages. This is the 2nd stage of the training.

The model shows improved proficiency with the Catalan language.

**This is an instruction fine-tuned model proficient on the following tasks in Catalan**

- *Information extraction (suitable for RAG)*
- *Named Entity Recognition (NER)*
- *Translation from English to Catalan and Catalan to English*
- *Summarization - both short form and long form*
- *Sentiment analysis*

The model achieves a loss rate of 0.8528 on the validation dataset after two epochs.

**NOTE:** The model was trained for one epoch on the `train` split of dataset and after manual evaluation, I decided to go for another epoch.

The first epoch logs every 100 steps while the second epoch logs every 200 steps, but I am pasting the train and eval losses for both epochs bellow.

*The `train` split of the dataset was shuffled before the second epoch. The `test` split dataset is identical in both epochs without shuffling*


**Model developers** [Laurentiu Petrea](https://www.linkedin.com/in/laurentiupetrea/) based on Llama-3 from Meta.

**Model Architecture** CataLlama is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and direct preference optimisation (DPO) to align with human preferences for helpfulness and safety.

**License** The model uses the llama-3 license available at: [https://llama.meta.com/llama3/license](https://llama.meta.com/llama3/license)


## Benchmarks

| Benchmark          | Value  |
| ------------------ | ------ |
| MMLU 5 shot        | 55.28  |
| GSM8K cot 8 shot   | 51.63  |


### Use with transformers

See the snippet below for usage with Transformers:

**The model follows the same prompt template as Llama-3 Instruct**

```python
import transformers
import torch

model_id = "catallama/CataLlama-v0.1-Instruct-SFT"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Ei com estàs avui?"},
]

prompt = pipeline.tokenizer.apply_chat_template(
    messages, 
    tokenize=False, 
    add_generation_prompt=True
)

outputs = pipeline(
    prompt,
    max_new_tokens=1024,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)

print(outputs[0]["generated_text"][len(prompt):])
```

## Training procedure

The model was trained **with the same prompt template of Llama-3 Instruct**.

The model was trained for two epochs on **6x A100 80GB GPUs using DeepSpeed ZeRO** State-3 without CPU offloading.

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- distributed_type: multi-GPU
- num_devices: 6
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 2

### Training results

**Epoch 1**

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.0938        | 0.11  | 100  | 1.0779          |
| 1.0186        | 0.22  | 200  | 1.0209          |
| 1.0157        | 0.32  | 300  | 0.9808          |
| 0.9588        | 0.43  | 400  | 0.9489          |
| 0.9039        | 0.54  | 500  | 0.9244          |
| 0.9111        | 0.65  | 600  | 0.9086          |
| 0.8918        | 0.75  | 700  | 0.8961          |
| 0.8971        | 0.86  | 800  | 0.8886          |
| 0.8631        | 0.97  | 900  | 0.8846          |


**Epoch 2**

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.8002        | 0.22  | 200  | 0.8989          |
| 0.8068        | 0.43  | 400  | 0.8835          |
| 0.7722        | 0.65  | 600  | 0.8654          |
| 0.7805        | 0.86  | 800  | 0.8528          |


## Intended Use

**Note:** This model is not intended to beat benchmarks, but to demonstrate techniques for augmenting LLMs on new languages and preserve rare languages as part of our world heritage.

**Intended Use Cases** Llama 3 is intended for commercial and research use in English. Instruction tuned models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks.

**Out-of-scope** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Llama 3 Community License. Use in languages other than English**.

**Note: Developers may fine-tune Llama 3 models for languages beyond English provided they comply with the Llama 3 Community License and the Acceptable Use Policy.