Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+llama3-8b-supernova-spectrum-hermes-dpo.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+llama3-8b-supernova-spectrum-hermes-dpo.bf16.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,90 @@
+---
+language:
+- en
+library_name: transformers
+license: apache-2.0
+pipeline_tag: text-generation
+tags:
+- dpo
+- rlhf
+- trl
+- autoquant
+- gguf
+---
+
+# Llama3-8B-SuperNova-Spectrum-Hermes-DPO
+
+This model is a **DPO fine-tuned** version of my `DARE_TIES` merged Model [`yuvraj17/Llama3-8B-SuperNova-Spectrum-dare_ties`](https://huggingface.co/yuvraj17/Llama3-8B-SuperNova-Spectrum-dare_ties) on the [yuvraj17/chatml-OpenHermes2.5-dpo-binarized-alpha-2k](https://huggingface.co/datasets/yuvraj17/chatml-OpenHermes2.5-dpo-binarized-alpha-2k) dataset.
+
+## DPO (Direct Preference Optimization):
+
+Direct Preference Optimization (DPO) is a fine-tuning technique that focuses on aligning a model's responses with human preferences or ranking data without requiring reinforcement learning steps, like in RLHF. 
+
+<figure>
+
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/66137d95e8d2cda230ddcea6/kHcU5dkcSVqxEIWt_GRUB.png" width="1000" height="768">
+  <figcaption> DPO vs RLHF <a href="//arxiv.org/abs/2305.18290">Reference</a> </figcaption>
+
+</figure>
+
+## Training:
+
+- Trained on **1x A40s (48GB VRAM)** using the [HuggingFace TRL](https://huggingface.co/docs/trl/index).
+- **QLoRA**(`4-bit precision`) for 1 epoch
+  ```
+  # LoRA configuration
+  peft_config = LoraConfig(
+      r=32,
+      lora_alpha=16,
+      lora_dropout=0.05,
+      bias="none",
+      task_type="CAUSAL_LM",
+      target_modules=['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']
+  )
+  ```
+### Training Params
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- beta=0.1
+- num_devices: 1
+- gradient_accumulation_steps: 4
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 100
+- num_epochs: 1
+
+### Training Time = **1:57:00** hours
+
+### Weight & Biases Report
+
+[Report-Link](https://api.wandb.ai/links/my-sft-team/d211juao)
+
+## 💻 Usage
+
+```python
+!pip install -qU transformers accelerate
+
+from transformers import AutoTokenizer
+import transformers
+import torch
+
+model = "yuvraj17/Llama3-8B-SuperNova-Spectrum-Hermes-DPO"
+messages = [{"role": "user", "content": "What is a large language model?"}]
+
+tokenizer = AutoTokenizer.from_pretrained(model)
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+pipeline = transformers.pipeline(
+    "text-generation",
+    model=model,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
+
+outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
+print(outputs[0]["generated_text"])
+```
+
+## 🏆 Evaluation Scores
+
+Coming Soon

llama3-8b-supernova-spectrum-hermes-dpo.Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f26a7a303c5b1cc18f65f1aecefaec70e6ce3b2f551a4fb9c170655f078b68da
+size 4018917920

llama3-8b-supernova-spectrum-hermes-dpo.bf16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b4ce5eccac9eb4aa88a469e0c1701b08bd219256515c079aaf9a67eb242a4de
+size 16068891168