Description
This repo contains bf16 files of Nyxene-v2-11B. It feels like with the new models, 1% is no longer needed as in the previous version. And yes, new version. Again.
Model used
- berkeley-nest/Starling-LM-7B-alpha
- openaccess-ai-collective/DPOpenHermes-7B
- fblgit/fblgit/una-cybertron-7b-v2
- chargoddard/loyal-piano-m7-cdpo
Prompt template
The best one after further testing is this one:
<|system|>
Below is an instruction that describes a task. Write a response that appropriately completes the request.
<|user|>
{prompt}
<|assistant|>
The secret sauce
loyal-piano-cybertron-11B :
slices:
- sources:
- model: fblgit/una-cybertron-7b-v2
layer_range: [0, 24]
- sources:
- model: chargoddard/loyal-piano-m7-cdpo
layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16
Starling-DPOHermes-11B :
slices:
- sources:
- model: berkeley-nest/Starling-LM-7B-alpha
layer_range: [0, 24]
- sources:
- model: openaccess-ai-collective/DPOpenHermes-7B
layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16
Nyxene-11B :
slices:
- sources:
- model: loyal-piano-cybertron-11B
layer_range: [0, 48]
- model: Starling-NeuralHermes-11B
layer_range: [0, 48]
merge_method: slerp
base_model: loyal-piano-cybertron-11B
parameters:
t:
- filter: lm_head
value: [0.75]
- filter: embed_tokens
value: [0.75]
- filter: self_attn
value: [0.75, 0.25]
- filter: mlp
value: [0.25, 0.75]
- filter: layernorm
value: [0.5, 0.5]
- filter: modelnorm
value: [0.75]
- value: 0.5 # fallback for rest of tensors
dtype: bfloat16
I use mergekit for all the manipulation told here.
Thanks to the Undi95 for the original 11B mistral merge recipe.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
Metric | Value |
---|---|
Avg. | 67.84 |
AI2 Reasoning Challenge (25-Shot) | 67.41 |
HellaSwag (10-Shot) | 84.54 |
MMLU (5-Shot) | 65.26 |
TruthfulQA (0-shot) | 55.62 |
Winogrande (5-shot) | 79.56 |
GSM8k (5-shot) | 54.66 |
- Downloads last month
- 1,263
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for beberik/Nyxene-v2-11B
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard67.410
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard84.540
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard65.260
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard55.620
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard79.560
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard54.660