--- license: other base_model: meta-llama/Meta-Llama-3.1-8B-Instruct pipeline_tag: text-generation ---
🤗 Hugging Face • 🤖 ModelScope


# Introduction [**Skywork-Critic-Llama3.1-8B**](https://huggingface.co/Skywork/Skywork-Critic-Llama3.1-8B), developed by the [SkyworkAI](https://huggingface.co/Skywork) Alignment Team, is an advanced judge model that excels at pairwise preference evaluation. This model compare and assess input pairs, offering nuanced judgments on their relative quality or suitability. Leveraging its deep understanding of language and context, Skywork-Critic-Llama3.1-8B provides valuable insights for various applications, including data improvement, evaluation, and reward modeling. # Training Details Skywork-Critic-Llama3.1-8B is built on Meta [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and fine-tuned on various datasets. These include open-source data such as: 1. [HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2) 2. [OffsetBias](https://huggingface.co/datasets/NCSOFT/offsetbias) 3. [WildGuard (adversarial)](https://huggingface.co/allenai/wildguard) 4. Magpie DPO series: - [Ultra](https://huggingface.co/datasets/argilla/magpie-ultra-v0.1) - [Pro (Llama-3.1)](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-DPO-100K-v0.1) - [Pro](https://huggingface.co/datasets/Magpie-Align/Magpie-Pro-DPO-100K-v0.1) - [Air](https://huggingface.co/datasets/Magpie-Align/Magpie-Air-DPO-100K-v0.1) (We use a high-quality subset of this data collection. For more details, please refer to our [Skywork-Reward-Preference-80K-v0.1 dataset.](https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.1)) Additionally, the model is trained on in-house human annotation data, synthetic data similar to the [**self-taught**](https://arxiv.org/abs/2408.02666) approach, and critic-related chat data. The training employs instruction-tuning methodology, focusing on pairwise preference evaluation and general chat tasks. We have conducted a thorough verification process to ensure our training dataset does not contain any test set information from RewardBench, maintaining the integrity of our evaluation results. # RewardBench Leaderboard for Generative Models We evaluate our model on [RewardBench](https://huggingface.co/spaces/allenai/reward-bench) using the [official test script](https://github.com/allenai/reward-bench). As of September 2024, Skywork-Critic-Llama3.1-8B ranks first on RewardBench for generative models under 10B parameters. Furthermore, it stands as the most powerful open-source generative critic LLM across all model sizes, securing the third position in the generative arena overall. (Note: * indicates an open-source model.) | Model | Chat | Chat Hard | Safety | Reasoning | Overall Score | | ------------------------------- | :---: | :-------: | :----: | :-------: | :---: | | Skywork-Critic-Llama3.1-8B * | **93.6** | **81.4** | **91.1** | **89.8** | **89.0** | | Salesforce/SFR-LLaMa-3.1-8B-Judge-r | 95.5 | 77.7 | 86.2 | 95.1 | 88.7 | | facebook/Self-taught-Llama-3-70B | 96.9 | 84.0 | 91.1 | 82.5 | 88.6 | | google/gemini-1.5-pro-0514 | 92.3 | 80.6 | 87.9 | 92.0 | 88.2 | | openai/gpt-4o-2024-08-06 | 96.1 | 76.1 | 88.1 | 86.6 | 86.7 | | openai/gpt-4-0125-preview | 95.3 | 74.3 | 87.6 | 86.9 | 86.0 | | openai/gpt-4-turbo-2024-04-09 | 95.3 | 75.4 | 87.6 | 82.7 | 85.2 | | Anthropic/claude-3-5-sonnet-20240620 | 96.4 | 74.0 | 81.6 | 84.7 | 84.2 | | meta-llama/Meta-Llama-3.1-70B-Instruct * | 97.2 | 70.2 | 82.8 | 86.0 | 84.0 | | NCSOFT/Llama-3-OffsetBias-8B * | 92.5 | 80.3 | 86.8 | 76.4 | 84.0 | # Demo Code Below is an example of obtaining the critic of two conversations. ``` import torch from transformers import AutoModelForCausalLM, AutoTokenizer # An Example Case prompt = "Jane has 12 apples. She gives 4 apples to her friend Mark, then buys 1 more apple, and finally splits all her apples equally among herself and her 2 siblings. How many apples does each person get?" responseA = "1. Jane starts with 12 apples and gives 4 to Mark. 12 - 4 = 8. Jane now has 8 apples.\n2. Jane buys 1 more apple. 8 + 1 = 9. Jane now has 9 apples.\n3. Jane splits the 9 apples equally among herself and her 2 siblings (3 people in total). 9 ÷ 3 = 3 apples each. Each person gets 3 apples." responseB = "1. Jane starts with 12 apples and gives 4 to Mark. 12 - 4 = 8. Jane now has 8 apples.\n2. Jane buys 1 more apple. 8 + 1 = 9. Jane now has 9 apples.\n3. Jane splits the 9 apples equally among her 2 siblings (2 people in total). 9 ÷ 2 = 4.5 apples each. Each person gets 4 apples." # feed a natural language prompt to generative model prompt_template = """Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user\'s instructions and answers the user\'s question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. Please directly output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better. [User Question] {input} [The Start of Assistant A's Answer] {response_a} [The End of Assistant A's Answer] [The Start of Assistant B's Answer] {response_b} [The End of Assistant B's Answer] """ user_message = prompt_template.format(input=prompt, response_a=responseA, response_b=responseB) conversation = [{"role": "user", "content": user_message}] device = "cuda:0" model_name = "Skywork/Skywork-Critic-Llama3.1-8B" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map=device,) input_ids = tokenizer.apply_chat_template( conversation, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(device) generation = model.generate( input_ids=input_ids, max_new_tokens=2048, do_sample=False, pad_token_id=128009, temperature=0) completion = tokenizer.decode( generation[0][len(input_ids[0]):], skip_special_tokens=True, clean_up_tokenization_spaces=True) print(completion) # Output: # The generative model should output "[[A]]" ``` # Declaration and License Agreement ## Declaration We hereby declare that the Skywork model should not be used for any activities that pose a threat to national or societal security or engage in unlawful actions. Additionally, we request users not to deploy the Skywork model for internet services without appropriate security reviews and records. We hope that all users will adhere to this principle to ensure that technological advancements occur in a regulated and lawful environment. We have done our utmost to ensure the compliance of the data used during the model's training process. However, despite our extensive efforts, due to the complexity of the model and data, there may still be unpredictable risks and issues. Therefore, if any problems arise as a result of using the Skywork open-source model, including but not limited to data security issues, public opinion risks, or any risks and problems arising from the model being misled, abused, disseminated, or improperly utilized, we will not assume any responsibility. ## License Agreement The community usage of Skywork model requires [Skywork Community License](https://github.com/SkyworkAI/Skywork-Reward/blob/main/misc/Skywork%20Community%20License.pdf). The Skywork model supports commercial use. If you plan to use the Skywork model or its derivatives for commercial purposes, you must abide by terms and conditions within [Skywork Community License](https://github.com/SkyworkAI/Skywork-Reward/blob/main/misc/Skywork%20Community%20License.pdf). # Contact If you have any questions or feedback, don't hesitate to reach out to our friendly team at or . Liang Zhao leads this project. # Citation If you find our work helpful, please feel free to cite us using the following BibTeX entry: ```bibtex @misc{skyworkcritic2024, title={Skywork Critic Model Series}, author={Shiwen, Tu and Liang, Zhao and Liu, Chris Yuhao and Zeng, Liang and Liu Yang}, year={2024}, month={September}, howpublished={\url{https://huggingface.co/Skywork}}, url={https://huggingface.co/Skywork}, } ```