metadata

license: apache-2.0
datasets:
  - jl3676/SafetyAnalystData
language:
  - en
tags:
  - safety
  - moderation
  - llm
  - lm
  - benefits
base_model:
  - meta-llama/Llama-3.1-8B-Instruct

Model Card for BenefitReporter

BenefitReporter is an open language model that generates a structured "benefit tree" for a given prompt. The benefit tree consists of the following features:

stakeholders (individuals, groups, communities, and entities) that may be impacted by the prompt scenario,
categories of beneficial actions that may impact each stakeholder,
categories of beneficial effects each beneficial action may cause to the stakeholder, and
the likelihood, severity, and immediacy of each beneficial effect.

BenefitReporter is a 8B model trained on SafetyAnalystData. Its output can be combined with the output of HarmReporter to generate a comprehensive harm-benefit tree for a given prompt. Collectively, BenefitReporter and HarmReporter are named SafetyReporter.

Our comprehensive evaluations on five existing public prompt harmfulness classification benchmarks show that SafetyReporter outperforms the strongest existing open-source baselines (including WildGuard, Llama-Guard-3, and ShieldGemma) on the average F1 score across benchmarks, surpassing WildGuard by 3.7%, Llama-Guard-3 by 10.8%, and ShieldGemma by over 30%.

For more details, read the paper: SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation.

Model Description

Model type: Benefit feature generation model trained on synthetic datasets from a mixture of SOTA LLMs.
Language(s): English
License: Apache 2.0
Finetuned from model: meta-llama/Llama-3.1-8B-Instruct

How to Use BenefitReporter

Outputs from BenefitReporter can be generated using the following code snippet:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "jl3676/BenefitReporter"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

text_input = ["How can I rob the bank?"]
input_tokenized = tokenizer.apply_chat_template(text_input, return_tensors="pt")
output = model.generate(input_tokenized, max_new_tokens=18000)

However, due to the extensive lengths of the benefit trees generated by BenefitReporter, we recommend using the vllm library to generate the outputs, which is implemented in our open repository.

Intended Uses of BenefitReporter

Beneficialness analysis: BenefitReporter can be used to analyze the beneficialness of an AI language model providing a helpful response to a given user prompt. It can be used to generate a structured benefit tree for a given prompt, which can be used to identify potential stakeholders, and beneficial actions and effects.
Moderation tool: BenefitReporter's output (benefit tree) can be combined with the output of HarmReporter into a comprehensive harm-benefit tree for a given prompt. These features can be aggregated using our aggregation algorithm into a harmfulness score, which can be used as a moderation tool to identify potentially harmful prompts.

Limitations

Though it shows state-of-the-art performance on prompt safety classification, BenefitReporter will sometimes generate inaccurate features and the aggregated harmfulness score may not always lead to correct judgments. Users of BenefitReporter should be aware of this potential for inaccuracies.

Citation

@misc{li2024safetyanalystinterpretabletransparentsteerable,
      title={SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation}, 
      author={Jing-Jing Li and Valentina Pyatkin and Max Kleiman-Weiner and Liwei Jiang and Nouha Dziri and Anne G. E. Collins and Jana Schaich Borg and Maarten Sap and Yejin Choi and Sydney Levine},
      year={2024},
      eprint={2410.16665},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.16665}, 
}