Update README.md

README.md

@@ -19,7 +19,7 @@ FuseChat-3.0: Preference Optimization for Implicit Model Fusion
   <img src="FuseChat-3.0.png" width=70%/>
 </div>
 
-We present FuseChat-3.0, a series of models crafted to enhance performance by integrating the strengths of multiple source LLMs into more compact target LLMs. To achieve this fusion, we utilized four powerful source LLMs: Gemma-2-27B-It, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct. For the target LLMs, we employed three widely-used smaller models—Llama-3.1-8B-Instruct, Gemma-2-9B-It, and Qwen-2.5-7B-Instruct—along with two even more compact models—Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct. The implicit model fusion process involves a two-stage training pipeline comprising Supervised Fine-Tuning (SFT) to mitigate distribution discrepancies between target and source LLMs, and Direct Preference Optimization (DPO) for learning preferences from multiple source LLMs. The resulting FuseChat-3.0 models demonstrated substantial improvements in tasks related to general conversation, instruction following, mathematics, and coding. Notably, when Llama-3.1-8B-Instruct served as the target LLM, our fusion approach achieved an average improvement of 6.8 points across 14 benchmarks. Moreover, it showed significant improvements of 37.1 and 30.1 points on instruction-following test sets AlpacaEval-2 and Arena-Hard respectively. We have released the [FuseChat-3.0](https://huggingface.co/FuseAI) models on Huggingface, stay tuned for the forthcoming dataset and code.
+We present FuseChat-3.0, a series of models crafted to enhance performance by integrating the strengths of multiple source LLMs into more compact target LLMs. To achieve this fusion, we utilized four powerful source LLMs: Gemma-2-27B-It, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct. For the target LLMs, we employed three widely-used smaller models—Llama-3.1-8B-Instruct, Gemma-2-9B-It, and Qwen-2.5-7B-Instruct—along with two even more compact models—Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct. The implicit model fusion process involves a two-stage training pipeline comprising Supervised Fine-Tuning (SFT) to mitigate distribution discrepancies between target and source LLMs, and Direct Preference Optimization (DPO) for learning preferences from multiple source LLMs. The resulting FuseChat-3.0 models demonstrated substantial improvements in tasks related to general conversation, instruction following, mathematics, and coding. Notably, when Llama-3.1-8B-Instruct served as the target LLM, our fusion approach achieved an average improvement of 6.8 points across 14 benchmarks. Moreover, it showed significant improvements of 37.1 and 30.1 points on instruction-following test sets AlpacaEval-2 and Arena-Hard respectively. We have released the [FuseChat-3.0](https://huggingface.co/FuseAI) models and datasets on Huggingface.
 
 
 
@@ -81,12 +81,14 @@ The sampling parameters for different models are detailed in Table below.
 </table>
 
 ### Data Construction
+Unlike the original approach in [WRPO](https://arxiv.org/abs/2412.03187), which constructs preference pairs from target model responses and treats source model responses as additional positive samples, our research in mathematics and coding domains revealed that sampling from multiple source models yields more and higher-quality preference pair data. Based on this insight, FuseChat-3.0 leverages the best and worst response pairs generated by source models as preference pairs to optimize the target model. This refined approach not only preserves the core advantages of implicit model fusion but also results in a more streamlined and practical implementation, making it particularly well-suited for real-world applications within the open-source community.
+
 - **Instruction Following**: To assign RM scores to the five responses generated by each source model, we employed [ArmoRM](https://huggingface.co/RLHFlow/ArmoRM-Llama3-8B-v0.1) for annotation. We then divided the annotated data into SFT and DPO datasets using a 4:6 ratio. For the SFT phase, we selected the responses with the highest RM scores. During the DPO phase, we paired responses from the same source model, designating those with the highest RM scores as positive samples and those with the lowest RM scores as negative samples. We ensured that the RM score difference between the positive and negative samples in each pair ranged from 0.01 to 0.1.
 - **Mathematics**: We initially annotated the responses from all source models for correctness by comparing them with the gold labels and evaluating them using the RM scores provided by ArmoRM. We then strategically divided the dataset into SFT phase and DPO phase. In the SFT phase, we incorporated responses that were correct and had the highest RM scores. This selection ensured that the fine-tuning process was based on high-quality responses that aligned closely with the desired outcomes. For the DPO phase, we constructed paired samples from the same source model. The positive samples consisted of correct answers with the highest RM scores, while the negative samples were incorrect answers with the lowest RM scores. To ensure meaningful comparisons during optimization, we maintained an RM score differential between positive and negative pairs within the range of 0.01 to 0.1.
 - **Coding**: We employed a dual-scoring system comprising correctness scores and RM scores for coding evaluation. The correctness scores assessed whether the code passed both static analysis and test cases, ensuring functional accuracy. The RM scores were used for preference evaluation, gauging the quality of responses based on predefined criteria. During the SFT phase, we included responses that not only passed all test cases but also achieved the highest RM scores. This selection ensured that the model was fine-tuned on exemplary code that met both correctness and preference standards. In the DPO phase, we contrasted positive samples—high-scoring responses that passed the tests—with negative samples—low-scoring responses that failed the tests. This comparison aimed to optimize the model's ability to prefer higher-quality code during training. We excluded any instances where all model responses failed to meet the testing criteria. This exclusion was necessary to maintain the integrity of the evaluation process, as such cases did not provide meaningful data for assessing and improving the model's performance.
 - **Chinese**: We exclusively utilized responses sampled from Qwen-2.5-72B-Instruct during the SFT phase, due to its strong performance in the Chinese language.
 
-Our final dataset comprised 158,784 total entries, with 94,539 entries for the SFT phase and 64,245 preference pairs for the DPO phase. The overall composition of the datasets is shown below.
+Our final dataset comprised 158,667 total entries, with 94,539 entries for the SFT phase and 64,128 preference pairs for the DPO phase. The overall composition of the datasets is shown below.
 
 <table class="js-sort-table table hidden">
   <tr>
@@ -123,7 +125,7 @@ Our final dataset comprised 158,784 total entries, with 94,539 entries for the S
 
   <tr>
     <td><a href="https://huggingface.co/datasets/nvidia/OpenMathInstruct-2" target="_blank">OpenMathInstruct-2</a></td>
-    <td>58546</td>
+    <td>51803</td>
     <td>40188</td>
     <td>11615</td>
     <td>Mathematics</td>
@@ -139,9 +141,9 @@ Our final dataset comprised 158,784 total entries, with 94,539 entries for the S
 
   <tr>
     <td><a href="https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k" target="_blank">self-oss-instruct-sc2</a></td>
-    <td>13696</td>
+    <td>12892</td>
     <td>10160</td>
-    <td>2849</td>
+    <td>2732</td>
     <td>Coding</td>
   </tr>
 
@@ -163,9 +165,9 @@ Our final dataset comprised 158,784 total entries, with 94,539 entries for the S
 
   <tr>
     <td><strong>Total</strong></td>
-    <td>158784</td>
+    <td>158667</td>
     <td>94539</td>
-    <td>64245</td>
+    <td>64128</td>
     <td>All</td>
   </tr>
 
@@ -386,10 +388,12 @@ The evaluation results of five series fused models are as follows, showing that
 
 ## Citation
 ```
-@article{yang2024wrpo,
-title={Weighted-Reward Preference Optimization for Implicit Model Fusion},
-author={Ziyi Yang and Fanqi Wan and Longguang Zhong and Tianyuan Shi and Xiaojun Quan},
-journal={arXiv preprint arXiv:2412.03187},
-year={2024}
+@inproceedings{yang2025weightedreward,
+  title={Weighted-Reward Preference Optimization for Implicit Model Fusion},
+  author={Ziyi Yang and Fanqi Wan and Longguang Zhong and Tianyuan Shi and Xiaojun Quan},
+  booktitle={The Thirteenth International Conference on Learning Representations},
+  year={2025},
+  url={https://openreview.net/forum?id=fq24pEb8SL}
 }
 ```
+