---
library_name: transformers
license: mit
language:
- en
metrics:
- pearsonr
- spearmanr
- accuracy
base_model:
- meta-llama/Llama-3.1-8B-Instruct
pipeline_tag: text-generation
---

# Model Card for Llama-Prometheus
Llama-Prometheus is a English evaluation model introduced as part of the CIA Suite to assess multilingual Large Language Models (LLMs).

Llama-Prometheus is fine-tuned on the Feedback-Collection dataset using the same setup as [Prometheus 2](https://huggingface.co/prometheus-eval/prometheus-7b-v2.0), but using the [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) as the base model. All FFT models and LoRA weights part of CIA Suite are available [here](https://huggingface.co/collections/ai4bharat/cia-suite-66ea9a7e18a6c70bd8de27a1).


# Model Details

## Model Description

- **Model type:** Evaluator Language model
- **Language(s) (NLP):** English
- **Related Models:** [Hercule Models](https://huggingface.co/collections/ai4bharat/cia-suite-66ea9a7e18a6c70bd8de27a1)
- **Resources for more information:**
  - [Research paper](https://arxiv.org/abs/2410.13394)
  - [GitHub Repo](https://github.com/AI4Bharat/CIA)


## Prompt Format

We’ve developed wrapper functions and classes to make it easy to work with Hercule. Check them out on our [github repository](https://github.com/AI4Bharat/CIA) – we highly recommend using them!

If you only need to use the model for your specific use case, please follow the prompt format provided below.

### Reference Guided Direct Assessment
The model expects four input components: an evaluation instruction, a response to evaluate, a scoring rubric, and a reference answer. Use the prompt format provided below, ensuring that you include the instruction, response, reference answer, evaluation criteria, and a detailed score rubric for each score from 1 to 5.

After running inference, the output will include feedback and a score, separated by the phrase ```[RESULT]```.
```
###Task Description:
An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.
1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
3. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)\"
4. Please do not generate any other opening, closing, and explanations.

###The instruction to evaluate:
{instruction}

###Response to evaluate:
{response}

###Reference Answer (Score 5):
{reference_answer}

###Score Rubrics:
[{criteria}]
Score 1: {score1_rubric}
Score 2: {score2_rubric}
Score 3: {score3_rubric}
Score 4: {score4_rubric}
Score 5: {score5_rubric}

###Feedback: 
```

We use the same evaluation prompt as used in [Prometheus 2](https://huggingface.co/prometheus-eval/prometheus-7b-v2.0).

## Links for Reference

- **Repository**: https://github.com/AI4Bharat/CIA 
- **Paper**: https://arxiv.org/abs/2410.13394
- **Point of Contact**: sumanthd@cse.iitm.ac.in, safikhan@ai4bharat.org

# Citation


If you find the following model helpful, please consider citing our paper!

**BibTeX:**

```bibtex
@article{doddapaneni2024crosslingual,
  title   = {Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs},
  author  = {Sumanth Doddapaneni and Mohammed Safi Ur Rahman Khan and Dilip Venkatesh and Raj Dabre and Anoop Kunchukuttan and Mitesh M. Khapra},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2410.13394}
}
```