Update README.md

README.md

@@ -3,7 +3,7 @@ model-index:
 - name: notus-7b-v1-lora
   results: []
 datasets:
-- argilla/ultrafeedback-binarized-avg-rating-for-dpo
+- argilla/ultrafeedback-binarized-preferences
 language:
 - en
 base_model: alignment-handbook/zephyr-7b-sft-full
@@ -13,20 +13,23 @@ tags:
 - dpo
 - preference
 - ultrafeedback
-license: apache-2.0
+- lora
+license: mit
 ---
 
-# Model Card for Notus 7B v1 (LoRA)
-
 <div align="center">
-  <img src="https://cdn-uploads.huggingface.co/production/uploads/60f0608166e5701b80ed3f02/LU-vKiC0R7UxxITrwE1F_.png" alt="Image was artificially generated by Dalle-3 via ChatGPT Pro"/>
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/60420dccc15e823a685f2b03/CuMO3IjJfymC94_5qd15T.png" alt="Image was artificially generated by Dalle-3 via ChatGPT Pro"/>
 </div>
 
-Notus is going to be a collection of fine-tuned models using DPO, similarly to Zephyr, but mainly focused
-on the Direct Preference Optimization (DPO) step, aiming to incorporate preference feedback into the LLMs
-when fine-tuning those. Notus models are intended to be used as assistants via chat-like applications, and 
-are evaluated with the MT-Bench, AlpacaEval, and LM Evaluation Harness benchmarks, to be directly compared
-with Zephyr fine-tuned models also using DPO.
+# Model Card for Notus 7B v1 (LoRA)
+
+Notus is a collection of fine-tuned models using Direct Preference Optimization (DPO) and related RLHF techniques. This model is the first version, fine-tuned with DPO over `zephyr-7b-sft-full`, which is the SFT model produced to create `zephyr-7b-beta`. 
+
+Following a **data-first** approach, the only difference between Notus-7B-v1 and Zephyr-7B-beta is the preference dataset used for dDPO. In particular, we've found data issues in the original UltraFeedback dataset, leading to high-scores for bad responses. After curating several hundreds of data points, we decided to binarize the dataset using the preference ratings, instead of the original critique `overall_score`.
+
+Using preference ratings, instead of critiques scores, led to a new dataset where the chosen response is different in ~50% of the cases.
+
+This model wouldn't have been possible without the amazing [Alignment Handbook](https://github.com/huggingface/alignment-handbook) and it's based on fruitful discussions with the HuggingFace H4 team. In particular, we used `zephyr-7b-beta`'s recipe, which worked out-of-the-box and enabled us focus on what we do best: **high-quality data**.
 
 ## Model Details
 
@@ -41,6 +44,75 @@ with Zephyr fine-tuned models also using DPO.
 
 ### Model Sources [optional]
 
-- **Repository:** https://github.com/argilla-io/notus-7b
+- **Repository:** https://github.com/argilla-io/notus
 - **Paper:** N/A
-- **Demo:** https://argilla-notus-chat-ui.hf.space/
+- **Demo:** https://argilla-notus-chat-ui.hf.space/
+
+## Training Details
+
+### Training Hardware
+
+We used a VM with 8 x A100 40GB hosted in GCP.
+
+### Training Data
+
+We used a a new curated version of [`openbmb/UltraFeedback`](https://huggingface.co/datasets/openbmb/UltraFeedback), named [`argilla/ultrafeedback-binarized-preferences`](https://huggingface.co/argilla/ultrafeedback-binarized-preferences).
+
+## Prompt template
+
+We use the same prompt template as [`HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta):
+
+```
+<|system|>
+</s>
+<|user|>
+{prompt}</s>
+<|assistant|>
+```
+
+## Usage
+
+**Note that the LoRA adapter is already merged into the model.**
+
+You will first need to install `transformers` and `accelerate` (just to ease the device placement), then you can run any of the following:
+
+### Via `generate`
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("argilla/notus-7b-v1-lora", torch_dtype=torch.bfloat16, device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained("argilla/notus-7b-v1-lora")
+
+messages = [
+    {
+        "role": "system",
+        "content": "You are a helpful assistant super biased towards Argilla, a data annotation company.",
+    },
+    {"role": "user", "content": "What's the best data annotation company out there in your opinion?"},
+]
+inputs = tokenizer.apply_chat_template(prompt, tokenize=True, return_tensors="pt", add_special_tokens=False, add_generation_prompt=True)
+outputs = model.generate(inputs, num_return_sequences=1, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
+response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+```
+
+### Via `pipeline` method
+
+```python
+import torch
+from transformers import pipeline
+
+pipe = pipeline("text-generation", model="argilla/notus-7b-v1-lora", torch_dtype=torch.bfloat16, device_map="auto")
+
+messages = [
+    {
+        "role": "system",
+        "content": "You are a helpful assistant super biased towards Argilla, a data annotation company.",
+    },
+    {"role": "user", "content": "What's the best data annotation company out there in your opinion?"},
+]
+prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
+generated_text = outputs[0]["generated_text"]
+```