library_name: transformers
tags:
- verifcation
- policy_compliance
- factchecking
- instruction_following
base_model:
- akjindal53244/Llama-3.1-Storm-8B
NAVI verifiers (Nace Automated Verification Intelligence)
Model Details
Model Description
NAVI (Nace Automated Verification Intelligence) is a solution for policy alignment verification designed to review various types of text against documents and policies, and identify violating content. It is specifically optimized for enterprise applications requiring compliance verification for automated text generation. To push policy verification in the open-source community, we release NAVI-small-preview, an open-weights version of the model we have deployed on the platform. NAVI-small-preview is centered around verifying specifically assitant outputs against some policy documents. The full solution is available on NAVI platform.
- Developed by: Nace
- Model type: Policy Alignment Verifier
- Language(s) (NLP): English
- License: [More Information Needed]
- Finetuned from model: akjindal53244/Llama-3.1-Storm-8B
Uses
Direct Use
NAVI-small-preview is used for verifying compliance of assistant outputs with enterprise policy documents. It processes policies and identifies any contradictions, inconsistencies and violations.
Downstream Use
May be integrated into enterprise-level AI systems to automate compliance checks in diverse industries such as legal, finance, and retail.
Out-of-Scope Use
NAVI is not designed for general-purpose factuality verification or tasks unrelated to policy compliance.
Bias, Risks, and Limitations
While NAVI excels in policy compliance verification, it may face challenges with unrealistic policy scenarios or contexts outside its training data scope.
Recommendations
We recommend to use the model only as a generative classifier outputting the class label, any other outputs are not accounted for during training.
How to Get Started with the Model
The verifier takes the assistant response and document context as inputs to generate a classification label. There are three possible classes: Compliant, Noncompliant, and Irrelevant. For long enterprise documents, we recommend setting up chunk-based vector search for selecting most relevant chunks from the document. Below are the prompts and sample code to launch:
Training Details
Training Data
NAVI was trained on a mix of real-world policy documents and synthetic interactions between a user and an assistant, it includes diverse, realistic, and complex policy examples across multiple industries.
Training Procedure
Preprocessing [optional]
Policies are processed into embeddings for effective decision-making.
Training Hyperparameters
- Training regime: [More Information Needed]
Evaluation
Testing Data, Factors & Metrics
Testing Data
We have manually collected Policy Alignment Verification dataset consisting of across different use cases for evaluation. We open source the public subset of the dataset. Here we diclose the performance on the public subset, containing 125 examples across six industry-specific scenarios: AT&T, Airbnb, Cadence Bank, Delta Airlines, Verisk, and Walgreens.
Factors
Evaluation focuses on policy compliance within multi-policy, multi-document contexts.
Metrics
F1 score was used to measure performance, prioritizing detection of noncompliance cases.
Results
NAVI-small-preview achieved an F1 score of 86.8%, outperforming all evaluated alternatives except full-scale NAVI. We evaluate against general-purpose solutions like Claude and Open AI models, as well as some guardrails focusing on factchecking to demonstrate a clear distinction of policy verification from the more common factchecking. Full performance metrics:
| Model | F1 Score | Avg Latency (ms) |
|---|---|---|
| NAVI-small-preview | 86.8 | - |
| NAVI | 90.4 | 387.62 |
| AWS Bedrock Guardrail | 76.5 | 342.79 |
| Azure Groundedness | 71.2 | 232.71 |
| NeMo (GPT-4o) | 71.2 | 2669.68 |
| GPT-4o (few-shot) | 75.0 | 904.46 |
| Sonnet 3.5 (few-shot) | 75.5 | 2926.69 |
Summary
NAVI demonstrates clear advantages in both performance and efficiency, making it a robust solution for policy compliance verification.
Model Card Contact
[More Information Needed]