Create README.md

README.md

@@ -0,0 +1,27 @@
+---
+base_model: wenqiglantz/MistralTrinity-7B-slerp
+tags:
+- mistral
+- instruct
+- finetune
+- chatml
+- synthetic data
+- distillation
+- dpo
+- rlhf
+license: apache-2.0
+language:
+- en
+datasets:
+- mlabonne/chatml_dpo_pairs
+---
+
+# MistralTrinity-7B-slerp-dpo
+
+Inspired by @mlabonne's blog post [Fine-tune a Mistral-7b model with Direct Preference Optimization](https://towardsdatascience.com/fine-tune-a-mistral-7b-model-with-direct-preference-optimization-708042745aac), this model was fine-tuned with DPO (Direct Preference Optimization) on base model `MistralTrinity-7B-slerp`, which is a merged model for `mistralai/Mistral-7B-Instruct-v0.2` and `jan-hq/trinity-v1`, using the [mlabonne/chatml_dpo_pairs](https://huggingface.co/datasets/mlabonne/chatml_dpo_pairs) dataset. 
+
+The code to train this model is available on [Google Colab](https://colab.research.google.com/github/wenqiglantz/llmops/blob/main/Fine_tune_MistralTrinity_7B_slerp_with_DPO.ipynb) and [GitHub](https://github.com/wenqiglantz/llmops/blob/main/Fine_tune_MistralTrinity_7B_slerp_with_DPO.ipynb). 
+
+It required an A100 GPU for over an hour.
+
+Check out fine-tuning run details on [Weights & Biases](https://wandb.ai/wenqiglantz/huggingface/runs/sxbgd33f).