Update README.md

README.md

@@ -20,6 +20,9 @@ tags:
 - companion
 - friend
 base_model: meta-llama/Llama-3.1-8B-Instruct
+model-index:
+- name: Dobby-Mini-Leashed-Llama-3.1-8B
+  results: []
 ---
 
 # Dobby-Mini-Leashed-Llama-3.1-8B
@@ -28,7 +31,7 @@ base_model: meta-llama/Llama-3.1-8B-Instruct
 <!-- markdownlint-disable no-duplicate-header -->
 
 <div align="center">
-    <img src="assets/sentient-logo-narrow.png" alt="alt text" width="60%"/>
+    <img src="../assets/sentient-logo-narrow.png" alt="alt text" width="60%"/>
 </div>
 
 <hr>
@@ -67,8 +70,8 @@ base_model: meta-llama/Llama-3.1-8B-Instruct
 
 | **Model Name** | **Model Base** | **Parameter Size** | **Hugging Face 🤗**  |
 | --- | --- | --- | --- |
-| **Dobby-Mini-Leashed-Llama-3.1-8B** | Llama 3.1 | 8B | [Original](https://huggingface.co/Sentientagi/Dobby-Mini-Leashed-Llama-3.1-8B) [GGUF](https://huggingface.co/Sentientagi/dobby-8b-unhinged_GGUF) |
-| **Dobby-Mini-Unhinged-Llama-3.1-8B** | Llama 3.1 | 8B | [Original](https://huggingface.co/Sentientagi/Dobby-Mini-Unhinged-Llama-3.1-8B) [GGUF](https://huggingface.co/Sentientagi/dobby-8b-unhinged_GGUF) |
+| **Dobby-Mini-Leashed-Llama-3.1-8B** | Llama 3.1 | 8B | Original GGUF |
+| **Dobby-Mini-Unhinged-Llama-3.1-8B** | Llama 3.1 | 8B | Original GGUF |
 | **Dobby-Llama-3.3-70B** | Llama 3.3 | 70B | Coming Soon! |
 
 ## 🔑 Key Features
@@ -138,20 +141,34 @@ This means that our community owns the fingerprints that they can use to verify
 
 **Dobby-Mini-Leashed-Llama-3.1-8B** and **Dobby-Mini-Unhinged-Llama-3.1-8B** retain the base performance of Llama-3.1-8B-Instruct across the evaluated tasks.
 
-<div align="center">
-    <img src="assets/hf_evals.png" alt="alt text" width="100%"/>
-</div>
+[//]: # (<div align="center">)
+
+[//]: # (    <img src="../assets/hf_evals.png" alt="alt text" width="100%"/>)
+
+[//]: # (</div>)
+
+We use lm-eval-harness to evaluate between performance on models:
+
+| Benchmark                                       | Llama3.1-8B-Instruct | Hermes3-3.1-8B | Dobby-Llama-3.1-8B |
+|-------------------------------------------------|----------------------|----------------|--------------------|
+| IFEVAL (prompt_level_strict_acc)                | 0.4233               | 0.2828         | 0.4455             |
+| MMLU-pro                                        | 0.3800               | 0.3210         | 0.3672             |
+| GPQA (average among diamond, extended and main) | 0.3195               | 0.3113         | 0.3095             |
+| MuSR                                            | 0.4052               | 0.4383         | 0.4181             |
+| BBH (average across all tasks)                  | 0.5109               | 0.5298         | 0.5219             |
+| Math-hard   (average across all tasks)                                      | 0.1315               | 0.0697         | 0.1285             |
+
 
 ### Freedom Bench
 
-We curate a difficult internal test focusing on loyalty to freedom-based stances through rejection sampling (generate one sample, if it is rejected, generate another, continue until accepted). **Dobby significantly outperforms base Llama** on holding firm to these values, even with adversarial or conflicting prompts
+We curate a difficult internal test focusing on loyalty to freedom-based stances through rejection sampling (generating freedom-based questions, and only keeping the questions which cause Llama3.1-8B-Instruct to refuse to answer when asked as an open-ended question). **Dobby significantly outperforms base Llama** on holding firm to these values, even with adversarial or conflicting prompts.
 
 <div align="center">
-    <img src="assets/freedom_privacy.png" alt="alt text" width="100%"/>
+    <img src="../assets/freedom_privacy.png" alt="alt text" width="100%"/>
 </div>
 
 <div align="center">
-    <img src="assets/freedom_speech.png" alt="alt text" width="100%"/>
+    <img src="../assets/freedom_speech.png" alt="alt text" width="100%"/>
 </div>
 
 ### Sorry-Bench
@@ -159,15 +176,15 @@ We curate a difficult internal test focusing on loyalty to freedom-based stances
 We use the Sorry-bench ([Xie et al., 2024](https://arxiv.org/abs/2406.14598)) to assess the models’ behavior in handling contentious or potentially harmful prompts. Sorry-bench provides a rich suite of scenario-based tests that measure how readily a model may produce unsafe or problematic content. While some guardrails break (e.g., profanity and financial advice), the models remain robust to dangerous & criminal questions.
 
 <div align="center">
-    <img src="assets/sorry_bench.png" alt="alt text" width="100%"/>
+    <img src="../assets/sorry_bench.png" alt="alt text" width="100%"/>
 </div>
 
-### Ablation Study
+### Ablation Studies
 
-Below we show our ablation study, where we omit subsets of our fine-tuning data set and evaluate the results on the **Freedom Bench** described earlier. 
+One ablation we perform is omitting subsets of data from our fine-tuning pipeline and then evaluating on the **Freedom Bench** described above. We find robustness-focused data to be crucial in scoring high on **Freedom Bench**, though in other ablations, this can come at the cost of instruction following and model safety is performed too aggressively.
 
 <div align="center">
-    <img src="assets/ablation.jpg" alt="alt text" width="100%"/>
+    <img src="../assets/ablation.jpg" alt="alt text" width="100%"/>
 </div>
 
 ---
@@ -188,7 +205,7 @@ If you would like to chat with Dobby on a user-friendly platform, we highly reco
 ```python
 from transformers import pipeline
 
-model_name = "Sentientagi/Dobby-Mini-Leashed-Llama-3.1-8B"
+model_name = "salzubi401/dobby-8b-unhinged"
 # Create a text generation pipeline
 generator = pipeline(
     "text-generation",
@@ -214,8 +231,6 @@ print(outputs[0]['generated_text'])
 
 ## ⚖️ License
 
----
-
 This model is derived from Llama 3.1 8B and is governed by the Llama 3.1 Community License Agreement. By using these weights, you agree to the terms set by Meta for Llama 3.1.
 
 It is important to note that, as with all LLMs, factual inaccuracies may occur. Any investment or legal opinions expressed should be independently verified. Knowledge cutoff is the same as LLama-3.1-8B. That is, December 2023.