--- license: apache-2.0 language: - en - zh base_model: - Qwen/Qwen2.5-14B - Qwen/Qwen2.5-14B-Instruct - Qwen/Qwen2.5-14B-Instruct-1M - EVA-UNIT-01/EVA-Qwen2.5-14B-v0.2 - Azure99/Blossom-V6-14B - arcee-ai/Virtuoso-Small-v2 pipeline_tag: text-generation tags: - merge model-index: - name: Qwen2.5-14B-1M-YOYO-V3 results: - task: type: text-generation name: Text Generation dataset: name: IFEval (0-Shot) type: HuggingFaceH4/ifeval args: num_few_shot: 0 metrics: - type: inst_level_strict_acc and prompt_level_strict_acc value: 83.98 name: strict accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: BBH (3-Shot) type: BBH args: num_few_shot: 3 metrics: - type: acc_norm value: 49.47 name: normalized accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MATH Lvl 5 (4-Shot) type: hendrycks/competition_math args: num_few_shot: 4 metrics: - type: exact_match value: 53.55 name: exact match source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GPQA (0-shot) type: Idavidrein/gpqa args: num_few_shot: 0 metrics: - type: acc_norm value: 10.51 name: acc_norm source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MuSR (0-shot) type: TAUR-Lab/MuSR args: num_few_shot: 0 metrics: - type: acc_norm value: 11.10 name: acc_norm source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU-PRO (5-shot) type: TIGER-Lab/MMLU-Pro config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 46.74 name: accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 name: Open LLM Leaderboard --- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64e174e202fa032de4143324/CfIE4_oZgpNsNZyurjO7D.png) # Qwen2.5-14B-1M-YOYO-V3 This time, I not only released the model but also shared some model merging insights that might be even more valuable than the model itself. Let’s start by looking at the initial merge configuration (YAML): ```yaml merge_method: model_stock base_model: Qwen/Qwen2.5-14B models: - model: Qwen/Qwen2.5-14B-instruct - model: Qwen/Qwen2.5-14B-instruct-1M dtype: bfloat16 ``` Does it seem like there are no issues at all? However, merged models occasionally exhibit **uncontrollable outputs**, likely due to significant discrepancies between instruction-tuned models and base models. To address this, I first attempted to directly integrate a fine-tuned model with smaller divergence from the base model, such as **Virtuoso-Small-v2**. This gave rise to [Qwen2.5-14B-YOYO-latest-V2](https://huggingface.co/YOYO-AI/Qwen2.5-14B-YOYO-latest-V2). ```yaml merge_method: model_stock base_model: Qwen/Qwen2.5-14B models: - model: Qwen/Qwen2.5-14B-instruct - model: Qwen/Qwen2.5-14B-instruct-1M - model: arcee-ai/Virtuoso-Small-v2 dtype: bfloat16 name: Qwen2.5-14B-YOYO-latest-V2 ``` Although the uncontrollable output issue has been addressed, the model still lacks stability. Through practical experimentation, I found that first merging **"high-divergence"** models (significantly different from the base) into **"low-divergence"** models (closer to the base) using the [DELLA](https://arxiv.org/abs/2406.11617) method, then applying the [Model Stock](https://arxiv.org/abs/2403.19522) method, ultimately produces a model that is not only more stable but also achieves better performance. ## Key models used: *1. Low-divergence, high-performance models:* - Virtuoso-Small-v2 - Blossom-V6-14B *2. High-divergence, instruction-focused models:* - Qwen2.5-14B-instruct - Qwen2.5-14B-instruct-1M ## DELLA Merge Configuration: ```yaml models: - model: Qwen/Qwen2.5-14B-Instruct parameters: density: 1 weight: 1 lambda: 0.9 merge_method: della base_model: arcee-ai/Virtuoso-Small-v2 parameters: density: 1 weight: 1 lambda: 0.9 normalize: true int8_mask: true dtype: bfloat16 tokenizer_source: base name: Qwen2.5-14B-YOYO-della1 ``` ```yaml models: - model: Qwen/Qwen2.5-14B-Instruct-1M parameters: density: 1 weight: 1 lambda: 0.9 merge_method: della base_model: arcee-ai/Virtuoso-Small-v2 parameters: density: 1 weight: 1 lambda: 0.9 normalize: true int8_mask: true dtype: bfloat16 tokenizer_source: base name: Qwen2.5-14B-YOYO-della2 ``` ```yaml models: - model: Qwen/Qwen2.5-14B-Instruct parameters: density: 1 weight: 1 lambda: 0.9 merge_method: della base_model: Azure99/Blossom-V6-14B parameters: density: 1 weight: 1 lambda: 0.9 normalize: true int8_mask: true dtype: bfloat16 tokenizer_source: base name: Qwen2.5-14B-YOYO-della3 ``` ```yaml models: - model: Qwen/Qwen2.5-14B-Instruct-1M parameters: density: 1 weight: 1 lambda: 0.9 merge_method: della base_model: Azure99/Blossom-V6-14B parameters: density: 1 weight: 1 lambda: 0.9 normalize: true int8_mask: true dtype: bfloat16 tokenizer_source: base name: Qwen2.5-14B-YOYO-della4 ``` This approach yielded four variants: - `Qwen2.5-14B-YOYO-della1` - `Qwen2.5-14B-YOYO-della2` - `Qwen2.5-14B-YOYO-della3` - `Qwen2.5-14B-YOYO-della4` ## Base Model: To enhance base model roleplay and creative writing capabilities, I applied the same strategy: ```yaml models: - model: EVA-UNIT-01/EVA-Qwen2.5-14B-v0.2 parameters: density: 1 weight: 1 lambda: 0.9 merge_method: della base_model: Qwen/Qwen2.5-14B parameters: density: 1 weight: 1 lambda: 0.9 normalize: true int8_mask: true dtype: bfloat16 tokenizer_source: base name: EVA-Qwen2.5-14B-base ``` Next, I extended the context length using the SCE method: ```yaml merge_method: sce models: - model: EVA-Qwen2.5-14B-base base_model: Qwen/Qwen2.5-14B-Instruct-1M parameters: select_topk: 1 dtype: bfloat16 tokenizer_source: base normalize: true int8_mask: true name: Qwen2.5-14B-pro ``` ## Final Merge Step: ```yaml merge_method: model_stock base_model: Qwen2.5-14B-pro models: - model: Qwen2.5-14B-YOYO-della1 - model: Qwen2.5-14B-YOYO-della2 - model: Qwen2.5-14B-YOYO-della3 - model: Qwen2.5-14B-YOYO-della4 dtype: bfloat16 tokenizer_source: base int8_mask: true normalize: true name: Qwen2.5-14B-1M-YOYO-V3 ``` I hope this helps! # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/YOYO-AI__Qwen2.5-14B-1M-YOYO-V3-details) | Metric |Value| |-------------------|----:| |Avg. |42.56| |IFEval (0-Shot) |83.98| |BBH (3-Shot) |49.47| |MATH Lvl 5 (4-Shot)|53.55| |GPQA (0-shot) |10.51| |MuSR (0-shot) |11.10| |MMLU-PRO (5-shot) |46.74|