| datasets: | |
| - monology/pile-uncopyrighted | |
| - MiniLLM/pile-diff_samp-qwen_1.8B-qwen_104M-r0.5 | |
| language: | |
| - en | |
| library_name: transformers | |
| license: apache-2.0 | |
| metrics: | |
| - accuracy | |
| pipeline_tag: text-generation | |
| # MiniPLM-llama3.1-212M | |
| [paper](https://arxiv.org/abs/2410.17215) | [code](https://github.com/thu-coai/MiniPLM) | |
| **MiniPLM-llama3.1-212M** is a 212M model with the [LLaMA3.1 achitecture](https://arxiv.org/abs/2407.21783) pre-trained from scratch on [the Pile](https://huggingface.co/datasets/monology/pile-uncopyrighted) using the MiniPLM knowledge distillation framework with the [offcial Qwen1.5-1.8B](https://huggingface.co/Qwen/Qwen1.5-1.8B) as the teacher model. | |
| This model shows the flexibility of the MiniPLM framework in conducting knowledge distillation across model families. | |
| We also open-source the [pre-training corpus](https://huggingface.co/datasets/MiniLLM/pile-diff_samp-qwen_1.8B-qwen_104M-r0.5) refined by Difference Sampling in MiniPLM for reproducibility. | |
| <p align='left'> | |
| <img src="https://cdn-uploads.huggingface.co/production/uploads/624ac662102fcdff87be51b9/2BqT0NgkmIXYlktovw9kG.png" width="1000"> | |
| </p> | |
| ## Evaluation | |
| MiniPLM models achieves better performance given the same computation and scales well across model sizes: | |
| <p align='left'> | |
| <img src="https://cdn-uploads.huggingface.co/production/uploads/624ac662102fcdff87be51b9/EOYzajQcwQFT5PobqL3j0.png" width="1000"> | |
| </p> | |
| ## Baseline Models | |
| + [Conventional Pre-Training](https://huggingface.co/MiniLLM/Pretrain-LLama3.1-130M) | |
| ## Citation | |
| ```bibtex | |
| @article{miniplm, | |
| title={MiniPLM: Knowledge Distillation for Pre-Training Language Models}, | |
| author={Yuxian Gu and Hao Zhou and Fandong Meng and Jie Zhou and Minlie Huang}, | |
| journal={arXiv preprint arXiv:2410.17215}, | |
| year={2024} | |
| } | |
| ``` |