Text Generation
Transformers
Safetensors
English
llama
mergekit
Merge
shining-valiant
shining-valiant-2
cobalt
plum
valiant
valiant-labs
llama-3.1
llama-3.1-instruct
llama-3.1-instruct-8b
llama-3
llama-3-instruct
llama-3-instruct-8b
8b
math
math-instruct
science
physics
biology
chemistry
compsci
computer-science
engineering
technical
conversational
chat
instruct
Eval Results
text-generation-inference
Inference Endpoints
language: | |
- en | |
license: llama3.1 | |
library_name: transformers | |
tags: | |
- mergekit | |
- merge | |
- shining-valiant | |
- shining-valiant-2 | |
- cobalt | |
- plum | |
- valiant | |
- valiant-labs | |
- llama | |
- llama-3.1 | |
- llama-3.1-instruct | |
- llama-3.1-instruct-8b | |
- llama-3 | |
- llama-3-instruct | |
- llama-3-instruct-8b | |
- 8b | |
- math | |
- math-instruct | |
- science | |
- physics | |
- biology | |
- chemistry | |
- compsci | |
- computer-science | |
- engineering | |
- technical | |
- conversational | |
- chat | |
- instruct | |
base_model: | |
- meta-llama/Llama-3.1-8B-Instruct | |
- ValiantLabs/Llama3.1-8B-ShiningValiant2 | |
- ValiantLabs/Llama3.1-8B-Cobalt | |
pipeline_tag: text-generation | |
model_type: llama | |
model-index: | |
- name: sequelbox/Llama3.1-8B-PlumMath | |
results: | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: Winogrande (5-Shot) | |
type: Winogrande | |
args: | |
num_few_shot: 5 | |
metrics: | |
- type: acc | |
value: 72.38 | |
name: acc | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: MathQA (5-Shot) | |
type: MathQA | |
args: | |
num_few_shot: 5 | |
metrics: | |
- type: acc | |
value: 40.27 | |
name: acc | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: IFEval (0-Shot) | |
type: HuggingFaceH4/ifeval | |
args: | |
num_few_shot: 0 | |
metrics: | |
- type: inst_level_strict_acc and prompt_level_strict_acc | |
value: 22.42 | |
name: strict accuracy | |
source: | |
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=sequelbox/Llama3.1-8B-PlumMath | |
name: Open LLM Leaderboard | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: BBH (3-Shot) | |
type: BBH | |
args: | |
num_few_shot: 3 | |
metrics: | |
- type: acc_norm | |
value: 16.45 | |
name: normalized accuracy | |
source: | |
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=sequelbox/Llama3.1-8B-PlumMath | |
name: Open LLM Leaderboard | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: MATH Lvl 5 (4-Shot) | |
type: hendrycks/competition_math | |
args: | |
num_few_shot: 4 | |
metrics: | |
- type: exact_match | |
value: 3.93 | |
name: exact match | |
source: | |
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=sequelbox/Llama3.1-8B-PlumMath | |
name: Open LLM Leaderboard | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: GPQA (0-shot) | |
type: Idavidrein/gpqa | |
args: | |
num_few_shot: 0 | |
metrics: | |
- type: acc_norm | |
value: 9.06 | |
name: acc_norm | |
source: | |
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=sequelbox/Llama3.1-8B-PlumMath | |
name: Open LLM Leaderboard | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: MuSR (0-shot) | |
type: TAUR-Lab/MuSR | |
args: | |
num_few_shot: 0 | |
metrics: | |
- type: acc_norm | |
value: 8.98 | |
name: acc_norm | |
source: | |
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=sequelbox/Llama3.1-8B-PlumMath | |
name: Open LLM Leaderboard | |
- task: | |
type: text-generation | |
name: Text Generation | |
dataset: | |
name: MMLU-PRO (5-shot) | |
type: TIGER-Lab/MMLU-Pro | |
config: main | |
split: test | |
args: | |
num_few_shot: 5 | |
metrics: | |
- type: acc | |
value: 21.95 | |
name: accuracy | |
source: | |
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=sequelbox/Llama3.1-8B-PlumMath | |
name: Open LLM Leaderboard | |
# PlumMath | |
This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit). | |
## Merge Details | |
### Merge Method | |
This model was merged using the della merge method using [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) as a base. | |
### Models Merged | |
The following models were included in the merge: | |
* [ValiantLabs/Llama3.1-8B-ShiningValiant2](https://huggingface.co/ValiantLabs/Llama3.1-8B-ShiningValiant2) | |
* [ValiantLabs/Llama3.1-8B-Cobalt](https://huggingface.co/ValiantLabs/Llama3.1-8B-Cobalt) | |
### Configuration | |
The following YAML configuration was used to produce this model: | |
```yaml | |
merge_method: della | |
dtype: bfloat16 | |
parameters: | |
normalize: true | |
models: | |
- model: ValiantLabs/Llama3.1-8B-ShiningValiant2 | |
parameters: | |
density: 0.5 | |
weight: 0.3 | |
- model: ValiantLabs/Llama3.1-8B-Cobalt | |
parameters: | |
density: 0.5 | |
weight: 0.2 | |
base_model: meta-llama/Llama-3.1-8B-Instruct | |
``` | |
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) | |
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_sequelbox__Llama3.1-8B-PlumMath) | |
| Metric |Value| | |
|-------------------|----:| | |
|Avg. |13.80| | |
|IFEval (0-Shot) |22.42| | |
|BBH (3-Shot) |16.45| | |
|MATH Lvl 5 (4-Shot)| 3.93| | |
|GPQA (0-shot) | 9.06| | |
|MuSR (0-shot) | 8.98| | |
|MMLU-PRO (5-shot) |21.95| | |