---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
---

DarwinLM is an evolutionary structured pruning method for large language models. It builds upon an evolutionary search process, generating multiple offspring models in each generation through mutation, and selecting the fittest for survival.  This significantly reduces the computational costs of LLMs, especially for real-time applications.

**Paper**: [https://arxiv.org/pdf/2502.07780](https://arxiv.org/pdf/2502.07780)  
**Code**: https://github.com/IST-DASLab/DarwinLM  
**Models**: [DarwinLM-2.7B](https://huggingface.co/Shengkun/DarwinLM-2.7B), [DarwinLM-4.6B](https://huggingface.co/Shengkun/DarwinLM-4.6B), [DarwinLM-8.4B](https://huggingface.co/Shengkun/DarwinLM-8.4B)  
**Pruned Models without Post-training**: [DarwinLM-2.7B-Pruned](https://huggingface.co/Shengkun/DarwinLM-2.7B-Pruned), [DarwinLM-4.6B-Pruned](https://huggingface.co/Shengkun/DarwinLM-4.6B-Pruned), [DarwinLM-8.4B-Pruned](https://huggingface.co/Shengkun/DarwinLM-8.4B-Pruned)  

---

This repository contains the weights of DarwinLM, as introduced in our paper. 

```python
# Please add trust_remote_code=True as the repo includes custom code to load and run DarwinLM
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Shengkun/DarwinLM-2.7B-Pruned", trust_remote_code=True)
```

## Downstream Tasks

**2.7B**

| Method                     | Param. | SciQ | PIQA | WG   | ArcE | ArcC | HS   | LogiQA | BoolQ | Avg  |
|----------------------------|--------|------|------|------|------|------|------|--------|-------|------|
| **Dense**                  | 6.7B   | 93.7 | 78.1 | 69.3 | 76.4 | 53.0 | 78.6 | 30.7   | 77.7  | 69.2 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |

**(Results for 4.6B and 8.4B)**

## Installation

```bash
conda env create -f environment.yml
conda activate darwinlm
```

## Database Preparation

```bash
# For llama-2-7B
bash scripts/ziplm_llama2-7B.sh
# ... other model examples
```

## Evolutionary Search

```bash
bash scripts/struct_prune_search.sh
```

## Post-Training

After pruning, you can further fine-tune the model with the [Fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset using the [llm-foundry](https://github.com/mosaicml/llm-foundry) repository. Refer to our paper for parameter settings.

## Evaluation
Install the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).

**Option 1: Using pre-trained weights:**

```bash
bash scripts/run_lmeval_hf.sh
```

**Option 2: Evaluating your searched structure:**

```bash
bash scripts/run_lmeval_config.sh
```


## Bibtex
```bibtex
@article{tang2025darwinlm,
  title={DarwinLM: Evolutionary Structured Pruning of Large Language Models},
  author={Tang, Shengkun and Sieberling, Oliver and Kurtic, Eldar and Shen, Zhiqiang and Alistarh, Dan},
  journal={arXiv preprint arXiv:2502.07780},
  year={2025}
}
```