README.md

---
language: en
license: apache-2.0
---

# Shears Model Card: shears-llama-7b-50-cs-heuristic-adapter

The heuristic adapter discovered from the [super-adapter](https://huggingface.co/IntelLabs/shears-llama-7b-50-cs-super-adapter) fine-tuned on sparsified LLaMA-7B with some commonsense reasoning datasets using Shears.

## Model Details

### Information

- **Model name:** shears-llama-7b-50-cs-heuristic-adapter
- **Base model:** [IntelLabs/Llama-1-7B-sparsity50](https://huggingface.co/IntelLabs/Llama-1-7B-sparsity50)
- **Sparsity:** 50%
- **Domain:** Commonsense
- **Subnetwork version:** Heuristic
- **NNCF Configuration:** [nncf_shears_llama.json](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Shears/nncf_config/nncf_shears_llama.json)

### Adapter Configuration

- **LoRA rank:** 32
- **LoRA alpha:** 64
- **LoRA target modules:** q_proj, k_proj, v_proj, up_proj, down_proj
- **LoRA rank search space:** [32, 24, 16] (for each LoRA module)

### Training Hyperparameters

- **Batch size:** 16
- **Learning rate:** 3e-4
- **Epoch:** 5

### Training Data

Unified commonsense reasoning dataset: [commonsense_170k.json](https://github.com/AGI-Edgerunners/LLM-Adapters/blob/main/ft-training_set/commonsense_170k.json).

### Evaluation Data
[BoolQ](https://github.com/AGI-Edgerunners/LLM-Adapters/blob/main/dataset/boolq/test.json), [PIQA](https://github.com/AGI-Edgerunners/LLM-Adapters/blob/main/dataset/piqa/test.json), [SIQA](https://github.com/AGI-Edgerunners/LLM-Adapters/blob/main/dataset/social_i_qa/test.json), [HellaSwag](https://github.com/AGI-Edgerunners/LLM-Adapters/blob/main/dataset/hellaswag/test.json), [WinoGrande](https://github.com/AGI-Edgerunners/LLM-Adapters/blob/main/dataset/winogrande/test.json), [ARC-e](https://github.com/AGI-Edgerunners/LLM-Adapters/blob/main/dataset/ARC-Easy/test.json), [ARC-c](https://github.com/AGI-Edgerunners/LLM-Adapters/blob/main/dataset/ARC-Challenge/test.json), [OBQA](https://github.com/AGI-Edgerunners/LLM-Adapters/blob/main/dataset/openbookqa/test.json).

## How to use

Use our modified PEFT library (apply [patch](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Shears/patches/peft-modifications-for-shears-inference-usage.patch)):
```bash
git clone https://github.com/huggingface/peft.git
cd peft && git checkout v0.5.0 && git apply --ignore-space-change --ignore-whitespace peft-modifications-for-shears-inference-usage.patch && pip install -e . && cd ..
```

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

def generate_prompt(instruction):
    return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request. 

                    ### Instruction:
                    {instruction}

                    ### Response:
                    """

base_model = AutoModelForCausalLM.from_pretrained("IntelLabs/Llama-1-7B-sparsity50")
model = PeftModel.from_pretrained(base_model, "IntelLabs/shears-llama-7b-50-cs-heuristic-adapter")
model.eval()

non_zero_params = sum([(param.data != 0).sum().item() for _, param in model.named_parameters()])
print(f"Number of all non-zero parameters: {non_zero_params}")

tokenizer = AutoTokenizer.from_pretrained("IntelLabs/Llama-1-7B-sparsity50")

instruction = "Please choose the correct answer to the question: A cactus stem is used to store\n\nAnswer1: fruit "
        "Answer2: liquid Answer3: food Answer4: spines\n\nAnswer format: answer1/answer2/answer3/answer4"
prompt = generate_prompt(instruction)
inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs["input_ids"].to(model.device)
with torch.no_grad():
    generation_output = model.generate(
        input_ids=input_ids,
        return_dict_in_generate=True,
        output_scores=True,
        max_new_tokens=256,
        use_cache=True,
        num_beams=4,
    )
    s = generation_output.sequences[0]
    output = tokenizer.decode(s)
print(output)

```

## Evaluation Results

| Model                | Sparsity  | BoolQ   | PIQA   | SIQA   | HellaSwag  | WinoG  | ARC-e  | ARC-c   | OBQA   | Average  |
|----------------------|-----------|---------|--------|--------|------------|--------|--------|---------|--------|----------|
| ChatGPT              | -         | 73.1    | 85.4   | 68.5   | 78.5       | 66.1   | 89.8   | 79.9    | 74.8   | 77.0     |
| LLaMA-7B-LoRA        | -         | 68.9    | 80.7   | 77.4   | 78.1       | 78.8   | 77.8   | 61.3    | 74.8   | 74.7     |
| [**LLaMA-7B-Shears**](https://huggingface.co/IntelLabs/shears-llama-7b-50-cs-heuristic-adapter)    | **50%**   | 67.3    | 79.1   | 77.5   | 73.3       | 77.7   | 74.4   | 57.9    | 72.8   | 72.5     |

## Model Sources

- **Repository:** [https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Shears](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Shears)
- **Paper:** [Shears: Unstructured Sparsity with Neural Low-rank Adapter Search](https://arxiv.org/abs/2404.10934)

## Citation

```bash
@article{munoz2024shears,
  title = {Shears: Unstructured Sparsity with Neural Low-rank Adapter Search},
  author={J. Pablo Munoz and Jinjie Yuan and Nilesh Jain},
  journal={The 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-2024)},
  year={2024}
}
```

## License

Apache-2.0