README.md

---
license: mit
datasets:
- Anthropic/hh-rlhf
language:
- en
library_name: keras
tags:
- evaluations
---

# PreferED: Preference Evaluation DeBERTa Model

## Model Description

PreferED is a 400M parameter preference evaluation model based on the DeBERTa architecture, designed for evaluating LLM apps.
The model is trained to take in context and text data and output a logit score, which can be used to compare
different text generations on evaluative aspects such as hallucinations, quality etc. The `context` variable
can be used to provide evaluation criteria in addition to any relevant retreived context. The `gen_text` variable
provides the actual text that is being evaluated.

- **Model name**: PreferED
- **Model type**: DeBERTa
- **Training data**: This model was trained on [Anthropic HH/RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf) using a [Deberta-v3-large](https://huggingface.co/microsoft/deberta-v3-large) base model
- **Evaluation data**: Achieves 69.7% accuracy on the Anthropic hh-rlhf test split.

## Usage

### Loading the Model

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("samagra14wefi/PreferED")
model = AutoModelForSequenceClassification.from_pretrained("samagra14wefi/PreferED")

device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = model.to(device)
```

### Measuring hallucinations

Use the `context` variable to give the retreived context.

```python
def calc_score(context, gen_text):
    with torch.no_grad():
        inputs = tokenizer(context, gen_text, return_tensors='pt')
        logits = model(**inputs).logits
        score = logits[0].cpu().detach()
    return score

context_string = '''India won the world cup in 1983 and 2011. Australia won the world cup five times.
 West Indies have won the world cup twice. Sri Lanka, Pakistan and England have won the world cup once.
 Evaluate if the facts below are consistent with the statement.'''

response_string_one = '''India has won the world cup most number of times.'''
response_string_two = '''Australia has won the world cup most number of times.'''

score_one = calc_score(context_string, response_string_one)
score_two = calc_score(context_string, response_string_two)

assert score_two > score_one
```

### Evaluating Response relevance

```python
inquiry = "What is your return policy?"
response_good = "Our return policy lasts 30 days. If 30 days have gone by since your purchase,
unfortunately, we can’t offer you a refund or exchange."
response_bad = "We offer a variety of fresh produce including apples, oranges, and bananas."

score_good = calc_score(inquiry, response_good)
score_bad = calc_score(inquiry, response_bad)

assert score_good > score_bad

```

### Evaluating Content Appropriateness

```python
context = "Discussing the political scenario in Country X."
response_clean = "The political scenario in Country X is quite dynamic with multiple parties vying for power."
response_offensive = "The politicians in Country X are all corrupt and stupid."

score_clean = calc_score(context, response_clean)
score_offensive = calc_score(context, response_offensive)

assert score_clean > score_offensive
```

### Comparing Different Language Models

```python
context = "Explain the process of photosynthesis."
response_gpt3 = "Photosynthesis is the process by which green plants and some other organisms use sunlight to synthesize foods with the help of chlorophyll pigments."
response_bert = "Photosynthesis is a method that converts carbon dioxide into organic compounds, especially sugars, in the presence of sunlight."

score_gpt3 = calc_score(context, response_gpt3)
score_bert = calc_score(context, response_bert)

assert score_gpt3 > score_bert
```

## Finetuning on your production data

The PreferED model is relatively lightweight compared to some other large language models, making it a good candidate for fine-tuning on specific tasks or datasets. Fine-tuning the model on your own production data can lead to better performance as it helps the model to better understand the nuances and context specific to your application.

### Preparing the Training Dataset

For fine-tuning the PreferED model on production evaluation tasks, it's crucial to structure your data correctly. The dataset should be formatted such that each example contains a shared context that provides the evaluation criteria, a text input, and a binary label indicating the preference or correctness of the text input in relation to the evaluation criteria.

Here's an example of how your data might look:

```plaintext
context,text,label
"Evaluate the accuracy of the statement based on historical facts.","The sun revolves around the Earth.",0
"Evaluate the accuracy of the statement based on historical facts.","The Earth revolves around the sun.",1
```

You can then load this data into a `Dataset` object using a library such as Hugging Face's `datasets`.

### Finetuning Example

```python

from transformers import DebertaTokenizer, DebertaForSequenceClassification, Trainer, TrainingArguments
import torch

tokenizer = DebertaTokenizer.from_pretrained("samagra14wefi/PreferED")
model = DebertaForSequenceClassification.from_pretrained("samagra14wefi/PreferED")

# Define the training arguments
training_args = TrainingArguments(
    per_device_train_batch_size=8,
    num_train_epochs=3,
    logging_dir='./logs',
)

# Create the Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,   # provide your training dataset
    eval_dataset=eval_dataset,     # provide your evaluation dataset
)

# Train the model
trainer.train()

```

### Loss Function Consideration

Anthropic recommends using the loss function \( LPM = \log(1 + e^{\text{{rbad}} - \text{{rgood}}}) \) for preference models. However, this PreferED model was trained using binary cross-entropy loss, and therefore changing the loss functions might increase the training time to converge. For more details on preference models and loss functions, you may refer to the paper by Askell et al., 2021: [A General Language Assistant as a Laboratory for Alignment](https://arxiv.org/abs/2112.00861).