WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models
davanstrien
Abstract
WalledEval is a comprehensive AI safety testing toolkit designed to evaluate large language models (LLMs). It accommodates a diverse range of models, including both open-weight and API-based ones, and features over 35 safety benchmarks covering areas such as multilingual safety, exaggerated safety, and prompt injections. The framework supports both LLM and judge benchmarking, and incorporates custom mutators to test safety against various text-style mutations such as future tense and paraphrasing. Additionally, WalledEval introduces WalledGuard, a new, small and performant content moderation tool, and SGXSTest, a benchmark for assessing exaggerated safety in cultural contexts. We make WalledEval publicly available at https://github.com/walledai/walledevalA.
Community
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal Behaviors (2024)
- CSRT: Evaluation and Analysis of LLMs using Code-Switching Red-Teaming Dataset (2024)
- Phi-3 Safety Post-Training: Aligning Language Models with a"Break-Fix"Cycle (2024)
- Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations (2024)
- SafeInfer: Context Adaptive Decoding Time Safety Alignment for Large Language Models (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Summarise it
