grounded-ai/phi3-rag-relevance-judge

Merged Model Performance

This repository contains our RAG relevance PEFT adapter model.

RAG Relevance Classification Metrics

Our merged model achieves the following performance on a binary classification task:

              precision    recall  f1-score   support

           0       0.74      0.77      0.75       100
           1       0.76      0.73      0.74       100

    accuracy                           0.75       200
   macro avg       0.75      0.75      0.75       200
weighted avg       0.75      0.75      0.75       200

Model Usage

For best results, we recommend starting with the following prompting strategy (and encourage tweaks as you see fit):

def format_input_classification(query, text):
    input = f"""
      You are comparing a reference text to a question and trying to determine if the reference text
  contains information relevant to answering the question. Here is the data:
      [BEGIN DATA]
      ************
      [Question]: {query}
      ************
      [Reference text]: {text}
      ************
      [END DATA]
  Compare the Question above to the Reference text. You must determine whether the Reference text
  contains information that can answer the Question. Please focus on whether the very specific
  question can be answered by the information in the Reference text.
  Your response must be single word, either "relevant" or "unrelated",
  and should not contain any text or characters aside from that word.
  "unrelated" means that the reference text does not contain an answer to the Question.
  "relevant" means the reference text contains an answer to the Question."""
    return input


text = format_input_classification("What is quanitzation?",
  "Quantization is a method to reduce the memory footprint")
messages = [
    {"role": "user", "content": text}
]

pipe = pipeline(
    "text-generation",
    model=base_model,
    model_kwargs={"attn_implementation": attn_implementation, "torch_dtype": torch.float16},
    tokenizer=tokenizer,
)

Comparison with Other Models

We compared our merged model's performance on the RAG Eval benchmark against several other state-of-the-art language models:

Model	Precision	Recall	F1
Our Merged Model	0.74	0.77	0.75
GPT-4	0.70	0.88	0.78
GPT-4 Turbo	0.68	0.91	0.78
Gemini Pro	0.61	1.00	0.76
GPT-3.5	0.42	1.00	0.59
Palm (Text Bison)	0.53	1.00	0.69
[1] Scores from arize/phoenix

As shown in the table, our merged model achieves a comparable score of 0.75, outperforming several other black box models.

We will continue to improve and fine-tune our merged model to achieve even better performance across various benchmarks and tasks.

Citations: [1] https://docs.arize.com/phoenix/evaluation/how-to-evals/running-pre-tested-evals/retrieval-rag-relevance