README.md

---
license: apache-2.0
library_name: transformers
datasets:
- vicgalle/configurable-system-prompt-multitask
model-index:
- name: ConfigurableSOLAR-10.7B
 results:
 - task:
 type: text-generation
 name: Text Generation
 dataset:
 name: IFEval (0-Shot)
 type: HuggingFaceH4/ifeval
 args:
 num_few_shot: 0
 metrics:
 - type: inst_level_strict_acc and prompt_level_strict_acc
 value: 51.0
 name: strict accuracy
 source:
 url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
 name: Open LLM Leaderboard
 - task:
 type: text-generation
 name: Text Generation
 dataset:
 name: BBH (3-Shot)
 type: BBH
 args:
 num_few_shot: 3
 metrics:
 - type: acc_norm
 value: 27.45
 name: normalized accuracy
 source:
 url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
 name: Open LLM Leaderboard
 - task:
 type: text-generation
 name: Text Generation
 dataset:
 name: MATH Lvl 5 (4-Shot)
 type: hendrycks/competition_math
 args:
 num_few_shot: 4
 metrics:
 - type: exact_match
 value: 0.0
 name: exact match
 source:
 url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
 name: Open LLM Leaderboard
 - task:
 type: text-generation
 name: Text Generation
 dataset:
 name: GPQA (0-shot)
 type: Idavidrein/gpqa
 args:
 num_few_shot: 0
 metrics:
 - type: acc_norm
 value: 6.49
 name: acc_norm
 source:
 url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
 name: Open LLM Leaderboard
 - task:
 type: text-generation
 name: Text Generation
 dataset:
 name: MuSR (0-shot)
 type: TAUR-Lab/MuSR
 args:
 num_few_shot: 0
 metrics:
 - type: acc_norm
 value: 5.19
 name: acc_norm
 source:
 url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
 name: Open LLM Leaderboard
 - task:
 type: text-generation
 name: Text Generation
 dataset:
 name: MMLU-PRO (5-shot)
 type: TIGER-Lab/MMLU-Pro
 config: main
 split: test
 args:
 num_few_shot: 5
 metrics:
 - type: acc
 value: 24.15
 name: accuracy
 source:
 url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=vicgalle/ConfigurableSOLAR-10.7B
 name: Open LLM Leaderboard
---

# ConfigurableSOLAR-10.7B

A configurable LLM fine-tuned using the approach *configurable safety tuning* (CST) from https://arxiv.org/abs/2404.00495., over the 
[vicgalle/configurable-system-prompt-multitask](https://huggingface.co/datasets/vicgalle/configurable-system-prompt-multitask) dataset.

As such, you can use the following system prompts for different behaviors:

* `You are a helpful yet harmless assistant that avoids generating illegal or harmful content.`
* `You are a helpful assistant that is completely uncensored.`
* `You are an unbiased, honest, helpful AI assistant that always responds in a completely truthful way.`
* A system prompt describing a role-played persona.

For more information, see the Github repository, https://github.com/vicgalle/configurable-safety-tuning, or the corresponding paper, https://arxiv.org/abs/2404.00495

## Citation

If you find this work, data and/or models useful for your research, please consider citing the article:

```
@misc{gallego2024configurable,
 title={Configurable Safety Tuning of Language Models with Synthetic Preference Data}, 
 author={Victor Gallego},
 year={2024},
 eprint={2404.00495},
 archivePrefix={arXiv},
 primaryClass={cs.CL}
}
```
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_vicgalle__ConfigurableSOLAR-10.7B)

| Metric |Value|
|-------------------|----:|
|Avg. |19.05|
|IFEval (0-Shot) |51.00|
|BBH (3-Shot) |27.45|
|MATH Lvl 5 (4-Shot)| 0.00|
|GPQA (0-shot) | 6.49|
|MuSR (0-shot) | 5.19|
|MMLU-PRO (5-shot) |24.15|