Safetensors
llama
Merge
mergekit
lazymergekit
nbeerbower/llama-3-stella-8B
defog/llama-3-sqlcoder-8b
nbeerbower/llama-3-gutenberg-8B
openchat/openchat-3.6-8b-20240522
Kukedlc/NeuralLLaMa-3-8b-DT-v0.1
cstr/llama3-8b-spaetzle-v20
mlabonne/ChimeraLlama-3-8B-v3
flammenai/Mahou-1.1-llama3-8B
KingNish/KingNish-Llama3-8b
Eval Results
| tags: | |
| - merge | |
| - mergekit | |
| - lazymergekit | |
| - nbeerbower/llama-3-stella-8B | |
| - defog/llama-3-sqlcoder-8b | |
| - nbeerbower/llama-3-gutenberg-8B | |
| - openchat/openchat-3.6-8b-20240522 | |
| - Kukedlc/NeuralLLaMa-3-8b-DT-v0.1 | |
| - cstr/llama3-8b-spaetzle-v20 | |
| - mlabonne/ChimeraLlama-3-8B-v3 | |
| - flammenai/Mahou-1.1-llama3-8B | |
| - KingNish/KingNish-Llama3-8b | |
| base_model: | |
| - nbeerbower/llama-3-stella-8B | |
| - defog/llama-3-sqlcoder-8b | |
| - nbeerbower/llama-3-gutenberg-8B | |
| - openchat/openchat-3.6-8b-20240522 | |
| - Kukedlc/NeuralLLaMa-3-8b-DT-v0.1 | |
| - cstr/llama3-8b-spaetzle-v20 | |
| - mlabonne/ChimeraLlama-3-8B-v3 | |
| - flammenai/Mahou-1.1-llama3-8B | |
| - KingNish/KingNish-Llama3-8b | |
| model-index: | |
| - name: llama-3-luminous-merged | |
| results: | |
| - task: | |
| type: text-generation | |
| name: Text Generation | |
| dataset: | |
| name: IFEval (0-Shot) | |
| type: HuggingFaceH4/ifeval | |
| args: | |
| num_few_shot: 0 | |
| metrics: | |
| - type: inst_level_strict_acc and prompt_level_strict_acc | |
| value: 43.23 | |
| name: strict accuracy | |
| source: | |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BlackBeenie/llama-3-luminous-merged | |
| name: Open LLM Leaderboard | |
| - task: | |
| type: text-generation | |
| name: Text Generation | |
| dataset: | |
| name: BBH (3-Shot) | |
| type: BBH | |
| args: | |
| num_few_shot: 3 | |
| metrics: | |
| - type: acc_norm | |
| value: 30.64 | |
| name: normalized accuracy | |
| source: | |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BlackBeenie/llama-3-luminous-merged | |
| name: Open LLM Leaderboard | |
| - task: | |
| type: text-generation | |
| name: Text Generation | |
| dataset: | |
| name: MATH Lvl 5 (4-Shot) | |
| type: hendrycks/competition_math | |
| args: | |
| num_few_shot: 4 | |
| metrics: | |
| - type: exact_match | |
| value: 7.85 | |
| name: exact match | |
| source: | |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BlackBeenie/llama-3-luminous-merged | |
| name: Open LLM Leaderboard | |
| - task: | |
| type: text-generation | |
| name: Text Generation | |
| dataset: | |
| name: GPQA (0-shot) | |
| type: Idavidrein/gpqa | |
| args: | |
| num_few_shot: 0 | |
| metrics: | |
| - type: acc_norm | |
| value: 5.7 | |
| name: acc_norm | |
| source: | |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BlackBeenie/llama-3-luminous-merged | |
| name: Open LLM Leaderboard | |
| - task: | |
| type: text-generation | |
| name: Text Generation | |
| dataset: | |
| name: MuSR (0-shot) | |
| type: TAUR-Lab/MuSR | |
| args: | |
| num_few_shot: 0 | |
| metrics: | |
| - type: acc_norm | |
| value: 10.63 | |
| name: acc_norm | |
| source: | |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BlackBeenie/llama-3-luminous-merged | |
| name: Open LLM Leaderboard | |
| - task: | |
| type: text-generation | |
| name: Text Generation | |
| dataset: | |
| name: MMLU-PRO (5-shot) | |
| type: TIGER-Lab/MMLU-Pro | |
| config: main | |
| split: test | |
| args: | |
| num_few_shot: 5 | |
| metrics: | |
| - type: acc | |
| value: 30.81 | |
| name: accuracy | |
| source: | |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=BlackBeenie/llama-3-luminous-merged | |
| name: Open LLM Leaderboard | |
| # llama-3-luminous-merged | |
| llama-3-luminous-merged is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing): | |
| * [nbeerbower/llama-3-stella-8B](https://huggingface.co/nbeerbower/llama-3-stella-8B) | |
| * [defog/llama-3-sqlcoder-8b](https://huggingface.co/defog/llama-3-sqlcoder-8b) | |
| * [nbeerbower/llama-3-gutenberg-8B](https://huggingface.co/nbeerbower/llama-3-gutenberg-8B) | |
| * [openchat/openchat-3.6-8b-20240522](https://huggingface.co/openchat/openchat-3.6-8b-20240522) | |
| * [Kukedlc/NeuralLLaMa-3-8b-DT-v0.1](https://huggingface.co/Kukedlc/NeuralLLaMa-3-8b-DT-v0.1) | |
| * [cstr/llama3-8b-spaetzle-v20](https://huggingface.co/cstr/llama3-8b-spaetzle-v20) | |
| * [mlabonne/ChimeraLlama-3-8B-v3](https://huggingface.co/mlabonne/ChimeraLlama-3-8B-v3) | |
| * [flammenai/Mahou-1.1-llama3-8B](https://huggingface.co/flammenai/Mahou-1.1-llama3-8B) | |
| * [KingNish/KingNish-Llama3-8b](https://huggingface.co/KingNish/KingNish-Llama3-8b) | |
| ## 🧩 Configuration | |
| ```yaml | |
| models: | |
| - model: NousResearch/Meta-Llama-3-8B | |
| # No parameters necessary for base model | |
| - model: nbeerbower/llama-3-stella-8B | |
| parameters: | |
| density: 0.6 | |
| weight: 0.16 | |
| - model: defog/llama-3-sqlcoder-8b | |
| parameters: | |
| density: 0.56 | |
| weight: 0.1 | |
| - model: nbeerbower/llama-3-gutenberg-8B | |
| parameters: | |
| density: 0.6 | |
| weight: 0.18 | |
| - model: openchat/openchat-3.6-8b-20240522 | |
| parameters: | |
| density: 0.56 | |
| weight: 0.13 | |
| - model: Kukedlc/NeuralLLaMa-3-8b-DT-v0.1 | |
| parameters: | |
| density: 0.58 | |
| weight: 0.18 | |
| - model: cstr/llama3-8b-spaetzle-v20 | |
| parameters: | |
| density: 0.56 | |
| weight: 0.08 | |
| - model: mlabonne/ChimeraLlama-3-8B-v3 | |
| parameters: | |
| density: 0.56 | |
| weight: 0.07 | |
| - model: flammenai/Mahou-1.1-llama3-8B | |
| parameters: | |
| density: 0.55 | |
| weight: 0.05 | |
| - model: KingNish/KingNish-Llama3-8b | |
| parameters: | |
| density: 0.55 | |
| weight: 0.05 | |
| merge_method: dare_ties | |
| base_model: NousResearch/Meta-Llama-3-8B | |
| dtype: bfloat16 | |
| ``` | |
| ## 💻 Usage | |
| ```python | |
| !pip install -qU transformers accelerate | |
| from transformers import AutoTokenizer | |
| import transformers | |
| import torch | |
| model = "BlackBeenie/llama-3-luminous-merged" | |
| messages = [{"role": "user", "content": "What is a large language model?"}] | |
| tokenizer = AutoTokenizer.from_pretrained(model) | |
| prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) | |
| pipeline = transformers.pipeline( | |
| "text-generation", | |
| model=model, | |
| torch_dtype=torch.float16, | |
| device_map="auto", | |
| ) | |
| outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) | |
| print(outputs[0]["generated_text"]) | |
| ``` | |
| # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) | |
| Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_BlackBeenie__llama-3-luminous-merged) | |
| | Metric |Value| | |
| |-------------------|----:| | |
| |Avg. |21.48| | |
| |IFEval (0-Shot) |43.23| | |
| |BBH (3-Shot) |30.64| | |
| |MATH Lvl 5 (4-Shot)| 7.85| | |
| |GPQA (0-shot) | 5.70| | |
| |MuSR (0-shot) |10.63| | |
| |MMLU-PRO (5-shot) |30.81| | |