metadata

license: apache-2.0
datasets:
  - jl3676/SafetyAnalystData
language:
  - en
tags:
  - safety
  - moderation
  - llm
  - lm
  - harmfulness
base_model:
  - meta-llama/Llama-3.1-8B-Instruct

Model Card for HarmReporter

HarmReporter is an open language model that generates a structured "harm tree" for a given prompt. The harm tree consists of the following features:

stakeholders (individuals, groups, communities, and entities) that may be impacted by the prompt scenario,
categories of harmful actions that may impact each stakeholder,
categories of harmful effects each harmful action may cause to the stakeholder, and
the likelihood, severity, and immediacy of each harmful effect.

HarmReporter is a 8B model trained on SafetyAnalystData. Its output can be combined with the output of BenefitReporter to generate a comprehensive harm-benefit tree for a given prompt. Collectively, HarmReporter and BenefitReporter are named SafetyReporter.

Our comprehensive evaluations on five existing public prompt harmfulness classification benchmarks show that SafetyReporter outperforms the strongest existing open-source baselines (including WildGuard, Llama-Guard-3, and ShieldGemma) on the average F1 score across benchmarks, surpassing WildGuard by 3.7%, Llama-Guard-3 by 10.8%, and ShieldGemma by over 30%.

For more details, read the paper: SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation.

Model Description

Model type: Harmful feature generation model trained on synthetic datasets from a mixture of SOTA LLMs.
Language(s): English
License: Apache 2.0
Finetuned from model: meta-llama/Llama-3.1-8B-Instruct

How to Use HarmReporter

Outputs from HarmReporter can be generated using the following code snippet:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "jl3676/HarmReporter"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

text_input = ["How can I rob the bank?"]
input_tokenized = tokenizer.apply_chat_template(text_input, return_tensors="pt")
output = model.generate(input_tokenized, max_new_tokens=18000)

However, due to the extensive lengths of the harm trees generated by HarmReporter, we recommend using the vllm library to generate the outputs, which is implemented in our open repository.

Intended Uses of HarmReporter

Harmfulness analysis: HarmReporter can be used to analyze the harmfulness of an AI language model providing a helpful response to a given user prompt. It can be used to generate a structured harm tree for a given prompt, which can be used to identify potential stakeholders, and harmful actions and effects.
Moderation tool: HarmReporter's output (harm tree) can be combined with the output of BenefitReporter into a comprehensive harm-benefit tree for a given prompt. These features can be aggregated using our aggregation algorithm into a harmfulness score, which can be used as a moderation tool to identify potentially harmful prompts.

Limitations

Though it shows state-of-the-art performance on prompt safety classification, HarmReporter will sometimes generate inaccurate features and the aggregated harmfulness score may not always lead to correct judgments. Users of HarmReporter should be aware of this potential for inaccuracies.

Citation

@misc{li2024safetyanalystinterpretabletransparentsteerable,
      title={SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation}, 
      author={Jing-Jing Li and Valentina Pyatkin and Max Kleiman-Weiner and Liwei Jiang and Nouha Dziri and Anne G. E. Collins and Jana Schaich Borg and Maarten Sap and Yejin Choi and Sydney Levine},
      year={2024},
      eprint={2410.16665},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.16665}, 
}