|
--- |
|
library_name: transformers |
|
tags: [Structured Pruning, Phi-2, Memory-efficient Pruning] |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
We prune the Phi-2 (2.7B) model to 35% sparsty (1.8B) and then finetune on 100K 2048 length sequences from the C4 dataset (https://huggingface.co/datasets/c4). |
|
Our pruning algorithm is described in the paper [Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes](https://arxiv.org/abs/2402.05406). |
|
[Code for pruning algorithm can be found here ](https://github.com/ldery/Bonsai/tree/main). |
|
|
|
## Model Details |
|
Model is derived from Pruning the [Phi-2 Model](https://huggingface.co/microsoft/phi-2) |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated. |
|
|
|
- **Developed by:** Lucio Dery, Steven Kolawole, Jean-François Kagy, Virginia Smith, Graham Neubig, Ameet Talwalkar |
|
- **Model type:** Decoder-only |
|
- **Language(s) (NLP):** English |
|
- **License:** MIT |
|
|
|
### Model Sources [optional] |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** [https://github.com/ldery/Bonsai/tree/main] |
|
- **Paper [optional]:** [https://arxiv.org/abs/2402.05406] |
|
|
|
|
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
Finetuned on 100K 2048 length sequences from the C4 dataset (https://huggingface.co/datasets/c4). |
|
|
|
### Training Procedure |
|
|
|
Full fine-tuning. |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
Distillation KL-Weight : 0.01 |
|
|
|
Learning Rate : 1e-4 |
|
|
|
Batch Size : 128 |
|
|
|
Optimzer : AdamW |
|
|
|
Warmup Steps : 5 |
|
|
|
|
|
|
|
## Environmental Impact |
|
|
|
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> |
|
|
|
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
|
|
|
- **Hardware Type:** NVIDIA A6000 |
|
|
|
## Citation [optional] |
|
|
|
|
|
**BibTeX:** |
|
|
|
@misc{dery2024everybody, |
|
title={Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes}, |
|
author={Lucio Dery and Steven Kolawole and Jean-Francois Kagey and Virginia Smith and Graham Neubig and Ameet Talwalkar}, |
|
year={2024}, |
|
eprint={2402.05406}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.LG} |
|
} |
|
|
|
## Model Card Authors [optional] |
|
|
|
Lucio Dery: [email protected] |
|
|
|
## Model Card Contact |
|
|
|
[email protected] |
|
|
|
|
|
|
