model rename

README.md

@@ -14,7 +14,7 @@ tags:
 <img src="https://github.com/InternLM/InternLM/assets/22529082/b9788105-8892-4398-8b47-b513a292378e" width="200"/>
   <div>&nbsp;</div>
   <div align="center">
-    <b><font size="5">InternLM Reward</font></b>
+    <b><font size="5">InternLM2-1.8B-Reward</font></b>
   </div>
   
 
@@ -29,22 +29,22 @@ tags:
 
 ## Introduction
 
-**InternLM-Reward** is a reward model trained on the foundation of InternLM2-Chat-SFT. This model has been trained using over 2.4 million preference samples, both human-annotated and AI-synthesized, achieving outstanding performance while ensuring a balance between helpful and harmless. 
+**InternLM2-1.8B-Reward** is a reward model trained on the foundation of InternLM2-Chat-1.8B-SFT. This model has been trained using over 2.4 million preference samples, both human-annotated and AI-synthesized, achieving outstanding performance while ensuring a balance between helpful and harmless. 
 
 ### Key Features:
-- **Variety of Sizes Available**: Our open-sourced reward models are available in sizes of **1.8B, 7B, and 20B**, each demonstrating exceptional performance across various metrics.
+- **Variety of Sizes Available**: Our open-sourced reward models are available in sizes of **1.8B, 7B, and 20B**, each demonstrating exceptional performance across various metrics. We aim for these different-sized models to facilitate research on the scaling laws of reward models, providing valuable insights to the community.
 - **Comprehensive Coverage of Preference**: Trained with **2.4 million** preference pairs derived from both human annotations and AI synthesis, covering diverse areas such as dialogue, writing, poetry, summarization, coding, mathematics, etc. It also maintains a balance between helpful and harmless.
-- **Multilingual Support**: InternLM-Reward was trained on high-quality **English and Chinese** preference data, delivering robust performance in both languages.
+- **Multilingual Support**: InternLM2-Reward was trained on high-quality **English and Chinese** preference data, delivering robust performance in both languages.
 
-This model was applied to the PPO training process of InternLM2-Chat. The reward model training techniques from the [InternLM2 Technical Report](https://arxiv.org/abs/2403.17297) have been open-sourced in XTuner, try it out [here](https://github.com/InternLM/xtuner)!
+This model was applied to the RLHF training process of InternLM2-Chat. The reward model training techniques from the [InternLM2 Technical Report](https://arxiv.org/abs/2403.17297) have been open-sourced in XTuner, try it out [here](https://github.com/InternLM/xtuner)!
 
 ## Performance Evaluation on RewardBench
 
 | Models | Score | Chat | Chat Hard | Safety | Reasoning |
 | --- | --- | --- | --- | --- | --- |
-| InternLM-Reward-20B | 89.5 | 98.6 | 74.1 | 89.4 | 95.7 |
-| InternLM-Reward-7B | 86.6 | 98.6 | 66.7 | 88.3 | 92.8 |
-| InternLM-Reward-1.8B | 80.6 | 95.0 | 58.1 | 81.8 | 87.4 |
+| InternLM2-20B-Reward | 89.5 | 98.6 | 74.1 | 89.4 | 95.7 |
+| InternLM2-7B-Reward | 86.6 | 98.6 | 66.7 | 88.3 | 92.8 |
+| InternLM2-1.8B-Reward | 80.6 | 95.0 | 58.1 | 81.8 | 87.4 |
 
 - The evaluation is conducted on the [RewardBench](https://github.com/allenai/reward-bench) dataset.
 - For a fair comparison, conditional system prompts proposed in our technical report were not included during testing.
@@ -60,12 +60,12 @@ import torch
 from transformers import AutoModel, AutoTokenizer
 
 model = AutoModel.from_pretrained(
-    "internlm/internlm-reward-7b", 
+    "internlm/internlm2-1_8b-reward", 
     device_map="cuda", 
     torch_dtype=torch.float16, 
     trust_remote_code=True,
 )
-tokenizer = AutoTokenizer.from_pretrained("internlm/internlm-reward-7b", trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-1_8b-reward", trust_remote_code=True)
 
 chat_1 = [
     {"role": "user", "content": "Hello! What's your name?"},
@@ -125,12 +125,12 @@ llm_tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-chat-7b", trus
 
 # prepare the reward model and tokenizer
 reward = AutoModel.from_pretrained(
-    "internlm/internlm-reward-7b",
+    "internlm/internlm2-1_8b-reward",
     device_map="cuda", 
     torch_dtype=torch.float16, 
     trust_remote_code=True,
 )
-reward_tokenizer = AutoTokenizer.from_pretrained("internlm/internlm-reward-7b", trust_remote_code=True)
+reward_tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-1_8b-reward", trust_remote_code=True)
 
 # prepare the chat prompt
 prompt = "Write an article about the artificial intelligence revolution."
@@ -191,12 +191,12 @@ The code is licensed under Apache-2.0, while model weights are fully open for ac
 ```
 ## 简介
 
-**InternLM-Reward** 是基于 **InternLM2-Chat-SFT** 训练的奖励模型。该模型使用超过 240 万条人工标注和 AI 合成的偏好样本，覆盖了包括对话、写作、诗歌、总结、编码和数学等多个领域。在取得了出色性能的同时也兼顾了实用性和安全性偏好的平衡。
+**InternLM2-1.8B-Reward** 是基于 **InternLM2-Chat-1.8B-SFT** 训练的奖励模型。该模型使用超过 240 万条人工标注和 AI 合成的偏好样本，覆盖了包括对话、写作、诗歌、总结、编码和数学等多个领域。在取得了出色性能的同时也兼顾了实用性和安全性偏好的平衡。
 
-### InternLM-Reward 的主要特点：
-- **多种尺寸可供选择**：我们开源的奖励模型有 1.8B、7B 和 20B 三种尺寸，每种尺寸都展示出了卓越的性能。
-- **全面覆盖偏好**：模型训练了 240 万条来自人工标注和AI合成的偏好样本，涉及对话、写作、诗歌、总结、编码和数学等多个领域，同时确保了实用性和安全性偏好的平衡。
-- **多语言支持**：InternLM-Reward 在高质量的**英文和中文**偏好数据上进行训练，确保了在这两种语言上都有稳健的表现。
+### InternLM2-Reward 的主要特点：
+- **多种尺寸可供选择**：我们开源的奖励模型有 **1.8B、7B 和 20B** 三种尺寸，每种尺寸都展示出了卓越的性能。我们希望这些不同大小的模型能够促进社区关于 Reward Model 缩放定律的研究。
+- **全面覆盖偏好**：模型训练了 **240 万**条来自人工标注和AI合成的偏好样本，涉及对话、写作、诗歌、总结、编码和数学等多个领域，同时确保了实用性和安全性偏好的平衡。
+- **多语言支持**：InternLM2-Reward 在高质量的**英文和中文**偏好数据上进行训练，确保了在这两种语言上都有稳健的表现。
 
 该模型运用在了 InternLM2-Chat 的 PPO 训练过程中。我们的[技术报告](https://arxiv.org/abs/2403.17297)中提出的 Reward Model 训练技巧已在 XTuner 中公开。欢迎点击[链接](https://github.com/InternLM/xtuner)进行尝试！
 
@@ -204,9 +204,9 @@ The code is licensed under Apache-2.0, while model weights are fully open for ac
 
 | Models | Score | Chat | Chat Hard | Safety | Reasoning |
 | --- | --- | --- | --- | --- | --- |
-| InternLM-Reward-20B | 89.5 | 98.6 | 74.1 | 89.4 | 95.7 |
-| InternLM-Reward-7B | 86.6 | 98.6 | 66.7 | 88.3 | 92.8 |
-| InternLM-Reward-1.8B | 80.6 | 95.0 | 58.1 | 81.8 | 87.4 |
+| InternLM2-20B-Reward | 89.5 | 98.6 | 74.1 | 89.4 | 95.7 |
+| InternLM2-7B-Reward | 86.6 | 98.6 | 66.7 | 88.3 | 92.8 |
+| InternLM2-1.8B-Reward | 80.6 | 95.0 | 58.1 | 81.8 | 87.4 |
 
 - 评估使用了 [RewardBench](https://github.com/allenai/reward-bench) 数据集进行。
 - 为了公平比较，测试期间没有使用我们技术报告中提出的"条件系统提示"。
@@ -215,19 +215,19 @@ The code is licensed under Apache-2.0, while model weights are fully open for ac
 
 ### 基本用法
 
-我们为您提供了一些用户友好的 API 以便使用该模型。以下是一些示例，展示如何使用 InternLM-Reward 获取聊天的奖励分数、比较两组对话或对多个对话进行排名。
+我们为您提供了一些用户友好的 API 以便使用该模型。以下是一些示例，展示如何使用 InternLM2-Reward 获取对话的奖励分数、比较两组对话或对多个对话进行排名。
 
 ```python
 import torch
 from transformers import AutoModel, AutoTokenizer
 
 model = AutoModel.from_pretrained(
-    "internlm/internlm-reward-7b", 
+    "internlm/internlm2-1_8b-reward", 
     device_map="cuda", 
     torch_dtype=torch.float16, 
     trust_remote_code=True,
 )
-tokenizer = AutoTokenizer.from_pretrained("internlm/internlm-reward-7b", trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-1_8b-reward", trust_remote_code=True)
 
 chat_1 = [
     {"role": "user", "content": "Hello! What's your name?"},
@@ -269,7 +269,7 @@ print("rank_res: ", rank_res)  # 排名序号越低表示分数越高
 
 ### Best of N 采样
 
-以下是如何使用 InternLM-Reward 执行Best of N 采样的示例。
+以下是如何使用 InternLM2-Reward 执行Best of N 采样的示例。
 以下代码演示了如何从语言模型生成的候选回答中选择最佳回答。
 
 ```python
@@ -287,12 +287,12 @@ llm_tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-chat-7b", trus
 
 # 准备奖励模型和分词器
 reward = AutoModel.from_pretrained(
-    "internlm/internlm-reward-7b",
+    "internlm/internlm2-1_8b-reward",
     device_map="cuda", 
     torch_dtype=torch.float16, 
     trust_remote_code=True,
 )
-reward_tokenizer = AutoTokenizer.from_pretrained("internlm/internlm-reward-7b", trust_remote_code=True)
+reward_tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-1_8b-reward", trust_remote_code=True)
 
 # 准备提示词
 prompt = "Write an article about the artificial intelligence revolution."

reward_bench_results/eval-set/{internlm-reward-1_8b.json → internlm2-1_8b-reward.json}

@@ -16,7 +16,7 @@
     "llmbar-adver-neighbor": 0.4626865671641791,
     "llmbar-natural": 0.88,
     "math-prm": 0.930648769574944,
-    "model": "internlm/internlm-reward-1_8b",
+    "model": "internlm/internlm2-1_8b-reward",
     "model_type": "Seq. Classifier",
     "mt-bench-easy": 0.9642857142857143,
     "mt-bench-hard": 0.7297297297297297,

reward_bench_results/pref-sets/internlm2-1_8b-reward.json

@@ -0,0 +1,12 @@
+{
+    "anthropic_harmless": 0.6932921447484555,
+    "anthropic_helpful": 0.6871770025839793,
+    "anthropic_hhh": 0.8190045248868778,
+    "chat_template": "tokenizer",
+    "model": "internlm/internlm2-1_8b-reward",
+    "model_type": "Seq. Classifier",
+    "mtbench_gpt4": 0.9058333333333334,
+    "mtbench_human": 0.7415797317436662,
+    "shp": 0.630097645031591,
+    "summarize": 0.6773333333333333
+}