README.md

---
language:
- en
license: llama3
library_name: transformers
tags:
- orpo
- llama 3
- rlhf
- sft
base_model:
- meta-llama/Meta-Llama-3-70B
datasets:
- mlabonne/orpo-dpo-mix-40k
model-index:
- name: Llama-3-70B-Orpo-v0.1
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: IFEval (0-Shot)
      type: HuggingFaceH4/ifeval
      args:
        num_few_shot: 0
    metrics:
    - type: inst_level_strict_acc and prompt_level_strict_acc
      value: 20.49
      name: strict accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=dfurman/Llama-3-70B-Orpo-v0.1
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: BBH (3-Shot)
      type: BBH
      args:
        num_few_shot: 3
    metrics:
    - type: acc_norm
      value: 24.09
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=dfurman/Llama-3-70B-Orpo-v0.1
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MATH Lvl 5 (4-Shot)
      type: hendrycks/competition_math
      args:
        num_few_shot: 4
    metrics:
    - type: exact_match
      value: 13.52
      name: exact match
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=dfurman/Llama-3-70B-Orpo-v0.1
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GPQA (0-shot)
      type: Idavidrein/gpqa
      args:
        num_few_shot: 0
    metrics:
    - type: acc_norm
      value: 1.01
      name: acc_norm
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=dfurman/Llama-3-70B-Orpo-v0.1
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MuSR (0-shot)
      type: TAUR-Lab/MuSR
      args:
        num_few_shot: 0
    metrics:
    - type: acc_norm
      value: 16.28
      name: acc_norm
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=dfurman/Llama-3-70B-Orpo-v0.1
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU-PRO (5-shot)
      type: TIGER-Lab/MMLU-Pro
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 32.14
      name: accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=dfurman/Llama-3-70B-Orpo-v0.1
      name: Open LLM Leaderboard
---

# dfurman/Llama-3-70B-Orpo-v0.1

![](https://raw.githubusercontent.com/daniel-furman/sft-demos/main/assets/llama_3.jpeg)

This is an ORPO fine-tune of [meta-llama/Meta-Llama-3-70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B) on 2k samples of [mlabonne/orpo-dpo-mix-40k](https://huggingface.co/datasets/mlabonne/orpo-dpo-mix-40k).

It's a successful fine-tune that follows the ChatML template!

## 🔎 Application

This model uses a context window of 8k. It was trained with the ChatML template.

## 🏆 Evaluation

### Open LLM Leaderboard

| Model ID                                                                                                                                                                                                                         |   Average |   ARC |   HellaSwag | MMLU  |   TruthfulQA |  Winogrande |  GSM8K  |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------: | --------: | --------: | ---------: | --------: |  --------: |  --------: |
| [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) [📄](https://huggingface.co/datasets/open-llm-leaderboard/details_meta-llama__Meta-Llama-3-70B-Instruct)    |       77.88 |     71.42 |     85.69 |      80.06 |     61.81 |     82.87 |     85.44 |
| [**dfurman/Llama-3-70B-Orpo-v0.1**](https://huggingface.co/dfurman/Llama-3-70B-Orpo-v0.1) [📄](https://huggingface.co/datasets/open-llm-leaderboard/details_dfurman__Llama-3-70B-Orpo-v0.1)                     | **74.67** | **68.69** | **88.01** | **79.39** | **49.62** |     **85.48** |     **76.8** |
| [meta-llama/Meta-Llama-3-70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B) [📄](https://huggingface.co/datasets/open-llm-leaderboard/details_meta-llama__Meta-Llama-3-70B)                               |     73.96 |      68.77 |     87.98 |      79.23 |      45.56 |     85.32 |     76.88 |



## 📈 Training curves

You can find the experiment on W&B at [this address](https://wandb.ai/dryanfurman/huggingface/runs/ojsbud95/workspace?nw=nwuserdryanfurman).


## 💻 Usage

<details>

<summary>Setup</summary>

```python
!pip install -qU transformers accelerate bitsandbytes

from transformers import AutoTokenizer, BitsAndBytesConfig
import transformers
import torch

if torch.cuda.get_device_capability()[0] >= 8:
    !pip install -qqq flash-attn
    attn_implementation = "flash_attention_2"
    torch_dtype = torch.bfloat16
else:
    attn_implementation = "eager"
    torch_dtype = torch.float16

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch_dtype,
    bnb_4bit_use_double_quant=True,
)

model = "dfurman/Llama-3-70B-Orpo-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    model_kwargs={
        "torch_dtype": torch_dtype,
        "quantization_config": bnb_config,
        "device_map": "auto",
        "attn_implementation": attn_implementation,
    }
)
```

</details>

### Run

```python
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me a recipe for a spicy margarita."},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print("***Prompt:\n", prompt)

outputs = pipeline(prompt, max_new_tokens=1000, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print("***Generation:\n", outputs[0]["generated_text"][len(prompt):])
```

<details>

<summary>Output</summary>

```
"""
"""
```
</details>

# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_dfurman__Llama-3-70B-Orpo-v0.1)

|      Metric       |Value|
|-------------------|----:|
|Avg.               |17.92|
|IFEval (0-Shot)    |20.49|
|BBH (3-Shot)       |24.09|
|MATH Lvl 5 (4-Shot)|13.52|
|GPQA (0-shot)      | 1.01|
|MuSR (0-shot)      |16.28|
|MMLU-PRO (5-shot)  |32.14|