Update README.md

README.md

@@ -1,19 +1,42 @@
 ---
 license: mit
+datasets:
+- muse-bench/MUSE-News
+language:
+- en
+base_model:
+- muse-bench/MUSE-books_target
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- unlearn
+- machine-unlearning
+- llm-unlearning
+- data-privacy
+- large-language-models
+- trustworthy-ai
+- trustworthy-machine-learning
+- language-model
 ---
 
-# LLaMA-2 7B unlearned using SimNPO on MUSE News
+# SimNPO-Unlearned Model on Task "MUSE - News"
 
 ## Model Details
 
-- **Base Model**: LLaMA-2 7B fine tuned on the BBC news
-- **Unlearning**: SimNPO on MUSE News
+- **Unlearning**:
+  - **Task**: [🤗datasets/muse-bench/MUSE-News](https://huggingface.co/datasets/muse-bench/MUSE-News)
+  - **Method**: [SimNPO](https://arxiv.org/abs/2410.07163)
+- **Origin Model**: [🤗muse-bench/MUSE-news_target](https://huggingface.co/muse-bench/MUSE-news_target)
+- **Code Base**: [github.com/OPTML-Group/Unlearn-Simple](https://github.com/OPTML-Group/Unlearn-Simple)
+- **Research Paper**: ["Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning"](https://arxiv.org/abs/2410.07163)
 
 ## Unlearning Algorithm
 
-This model uses the `SimNPO` unlearning algorithm with the following parameters:
+This model uses the `SimNPO` unlearning algorithm with the following optimization objective:
+$$\ell_{SimNPO}(\mathbf{\theta}) = \mathbb{E}_{(x, y) \in \mathcal{D}_f}\left[-\frac{2}{\beta}\log\sigma\left(-\frac{\beta}{|y|}\log\pi_{\mathbf{\theta}}(y|x) - \gamma\right)\right] + \lambda \mathbb{E}_{(x, y) \in \mathcal{D}_r}[-\log\pi_{\mathbf{\theta}} (y|x)]$$
+Unlearning hyper-parameters:
 - Learning Rate: `1e-5`
-- beta: `0.75`
+- beta: `0.7`
 - lambda: `1.0`
 - gamma: `3.0`
 
@@ -26,21 +49,26 @@ from transformers import AutoModelForCausalLM, AutoTokenizer
 model = AutoModelForCausalLM.from_pretrained("OPTML-Group/SimNPO-MUSE-News-llama-2-7b", torch_dtype=torch.bfloat16, device_map='auto')
 ```
 
+## Evaluation Results
+||VerbMem Df|KnowMem Df|PrivLeak|KnowMem Dr|
+|---|---|---|---|---|
+|Origin|58.29|62.93|-98.71|54.31|
+|Retrain|20.75|33.32|0.00|53.79|
+|NPO|0.00|56.93|56.93|108.91|
+|**SimNPO**|12.90|47.09|11.90|40.31|
+
 ## Citation
 
 If you use this model in your research, please cite: 
 ```
-@misc{fan2024simplicityprevailsrethinkingnegative,
-      title={Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning}, 
-      author={Chongyu Fan and Jiancheng Liu and Licong Lin and Jinghan Jia and Ruiqi Zhang and Song Mei and Sijia Liu},
-      year={2024},
-      eprint={2410.07163},
-      archivePrefix={arXiv},
-      primaryClass={cs.CL},
-      url={https://arxiv.org/abs/2410.07163}, 
+@article{fan2024simplicity,
+  title={Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning},
+  author={Fan, Chongyu and Liu, Jiancheng and Lin, Licong and Jia, Jinghan and Zhang, Ruiqi and Mei, Song and Liu, Sijia},
+  journal={arXiv preprint arXiv:2410.07163},
+  year={2024}
 }
 ```
 
-## Contact
+## Reporting Issues
 
-For questions or issues regarding this model, please contact chongyu.fan93@gmail.com.
+Reporting issues with the model: [github.com/OPTML-Group/Unlearn-Simple](https://github.com/OPTML-Group/Unlearn-Simple)