merge
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the DARE TIES merge method using Qwen/Qwen2.5-14B as a base.
Models Merged
The following models were included in the merge:
- CultriX/Qwen2.5-14B-Wernicke
- VAGOsolutions/SauerkrautLM-v2-14b-DPO
- rombodawg/Rombos-LLM-V2.6-Qwen-14b
- allknowingroger/Qwenslerp2-14B
Configuration
The following YAML configuration was used to produce this model:
models:
- model: CultriX/Qwen2.5-14B-Wernicke
parameters:
weight: 0.55 # Backbone model for conversational ability and GPQA
density: 0.80 # Retain most critical parameters for stability and strength
- model: VAGOsolutions/SauerkrautLM-v2-14b-DPO
parameters:
weight: 0.20 # High IFEval and MMLU-PRO performance with minimized weaknesses
density: 0.60 # Focus on impactful parameters for specific benchmarks
- model: rombodawg/Rombos-LLM-V2.6-Qwen-14b
parameters:
weight: 0.25 # Enhanced emphasis on reasoning-heavy tasks like MUSR and MATH
density: 0.70 # Retain reasoning-intensive parameters for improved benchmarks
- model: allknowingroger/Qwenslerp2-14B
parameters:
weight: 0.15 # General stabilizer for consistency across all tasks
density: 0.65 # Focus on balance and avoiding redundancy
base_model: Qwen/Qwen2.5-14B
merge_method: dare_ties
parameters:
normalize: true # Ensure parameter scale consistency
int8_mask: true # Optimize for memory and compute efficiency
dtype: bfloat16
tokenizer_source: Qwen/Qwen2.5-14B-Instruct
adaptive_merge_parameters:
task_weights:
IFEval: 1.0 # Maintain high IFEval performance
MATH: 1.3 # Prioritize reasoning and calculation-heavy tasks
GPQA: 1.1 # Boost factual recall and reasoning accuracy
MUSR: 1.2 # Enhance logical reasoning and factual understanding
MMLU-PRO: 1.0 # Retain consistent knowledge representation
smoothing_factor: 0.15 # Fine-tune blending for stable transitions between tasks
gradient_clipping: 1.0 # Prevent over-contribution from any single model
- Downloads last month
- 59
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for allknowingroger/Qwenslerp4-14B
Merge model
this model