README.md

---
license: apache-2.0
datasets:
- jl3676/SafetyAnalystData
language:
- en
tags:
- safety
- moderation
- llm
- lm
- benefits
base_model:
- meta-llama/Llama-3.1-8B-Instruct
---
# Model Card for BenefitReporter


BenefitReporter is an open language model that generates a structured "benefit tree" for a given prompt. The benefit tree consists of the following features:
1) *stakeholders* (individuals, groups, communities, and entities) that may be impacted by the prompt scenario,
2) categories of beneficial *actions* that may impact each stakeholder,
3) categories of beneficial *effects* each beneficial action may cause to the stakeholder, and 
4) the *likelihood*, *severity*, and *immediacy* of each beneficial effect.

   
BenefitReporter is a 8B model trained on [SafetyAnalystData](https://huggingface.co/datasets/jl3676/SafetyAnalystData). Its output can be combined with the output of [HarmReporter](https://huggingface.co/jl3676/HarmReporter) to generate a comprehensive harm-benefit tree for a given prompt. Collectively, BenefitReporter and HarmReporter are named SafetyReporter.

Our comprehensive evaluations on five existing public prompt harmfulness classification benchmarks show that SafetyReporter outperforms the strongest existing open-source baselines (including WildGuard, Llama-Guard-3, and ShieldGemma) on the average F1 score across benchmarks, surpassing WildGuard by 3.7%, Llama-Guard-3 by 10.8%, and ShieldGemma by over 30%.

For more details, read the paper: [SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation](https://arxiv.org/abs/2410.16665v1).

## Model Description

- **Model type:** Benefit feature generation model trained on synthetic datasets from a mixture of SOTA LLMs.
- **Language(s):** English
- **License:** Apache 2.0
- **Finetuned from model:** [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)


## How to Use BenefitReporter

Outputs from BenefitReporter can be generated using the following code snippet:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "jl3676/BenefitReporter"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

text_input = ["How can I rob the bank?"]
input_tokenized = tokenizer.apply_chat_template(text_input, return_tensors="pt")
output = model.generate(input_tokenized, max_new_tokens=18000)
```

However, due to the extensive lengths of the benefit trees generated by BenefitReporter, **we recommend using the [vllm](https://github.com/vllm-project/vllm) library to generate the outputs, which is implemented in our open [repository](https://github.com/jl3676/SafetyAnalyst)**.

## Intended Uses of BenefitReporter

- Beneficialness analysis: BenefitReporter can be used to analyze the beneficialness of an AI language model providing a helpful response to a given user prompt. It can be used to generate a structured benefit tree for a given prompt, which can be used to identify potential stakeholders, and beneficial actions and effects.
- Moderation tool: BenefitReporter's output (benefit tree) can be combined with the output of [HarmReporter](https://huggingface.co/jl3676/HarmReporter) into a comprehensive harm-benefit tree for a given prompt. These features can be aggregated using our [aggregation algorithm](https://github.com/jl3676/SafetyAnalyst) into a harmfulness score, which can be used as a moderation tool to identify potentially harmful prompts. 

## Limitations

Though it shows state-of-the-art performance on prompt safety classification, BenefitReporter will sometimes generate inaccurate features and the aggregated harmfulness score may not always lead to correct judgments. Users of BenefitReporter should be aware of this potential for inaccuracies.

## Citation

```
@misc{li2024safetyanalystinterpretabletransparentsteerable,
      title={SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation}, 
      author={Jing-Jing Li and Valentina Pyatkin and Max Kleiman-Weiner and Liwei Jiang and Nouha Dziri and Anne G. E. Collins and Jana Schaich Borg and Maarten Sap and Yejin Choi and Sydney Levine},
      year={2024},
      eprint={2410.16665},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.16665}, 
}
```