Add instruction for the sparse base model

README.md

@@ -12,12 +12,38 @@ The heuristic adapter discovered from the [super-adapter](https://huggingface.co
 ### Information
 
 - **Model name:** shears-llama-7b-50-cs-heuristic-adapter
-- **Base model:** [IntelLabs/shears-llama-7b-50-base](https://huggingface.co/IntelLabs/shears-llama-7b-50-base)
+- **Base model:** Sparsified [LLaMA-7B](https://huggingface.co/yahma/llama-7b-hf)
 - **Sparsity:** 50%
 - **Domain:** Commonsense
 - **Subnetwork version:** Heuristic
 - **NNCF Configuration:** [nncf_shears_llama.json](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Shears/nncf_config/nncf_shears_llama.json)
 
+### Sparsified Base Model
+
+Shears employs a simple but effective pruning approach [Wanda](https://arxiv.org/abs/2306.11695) to sparsify the language model, serving as the base model.
+Clone the [Wanda](https://github.com/locuslab/wanda) repo:
+
+```bash
+git clone https://github.com/locuslab/wanda.git && cd wanda && git checkout 8e8fc87 && cd ..
+```
+
+The command for unstructured sparsifying LLaMA-7B with Wanda, to achieve unstructured 50% sparsity:
+
+```bash
+python wanda/main.py \
+    --model yahma/llama-7b-hf \
+    --prune_method wanda \
+    --sparsity_ratio 0.5 \
+    --sparsity_type unstructured \
+    --save wanda_out \
+    --save_model shears-llama-7b-50-base
+```
+- `--model`: The identifier for the model on the Hugging Face model hub or local path.
+- `--sparsity_ratio`: Specifies the percentage of weights to be pruned.
+- `--save_model`: Specifies the directory where the pruned language model will be stored.
+
+Refer to our [repo](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Shears#setup) for the environment information to run this command.
+
 ### Adapter Configuration
 
 - **LoRA rank:** 32
@@ -61,14 +87,14 @@ def generate_prompt(instruction):
                     ### Response:
                     """
 
-base_model = AutoModelForCausalLM.from_pretrained("IntelLabs/shears-llama-7b-50-base")
+base_model = AutoModelForCausalLM.from_pretrained("shears-llama-7b-50-base")
 model = PeftModel.from_pretrained(base_model, "IntelLabs/shears-llama-7b-50-cs-heuristic-adapter")
 model.eval()
 
 non_zero_params = sum([(param.data != 0).sum().item() for _, param in model.named_parameters()])
 print(f"Number of all non-zero parameters: {non_zero_params}")
 
-tokenizer = AutoTokenizer.from_pretrained("IntelLabs/shears-llama-7b-50-base")
+tokenizer = AutoTokenizer.from_pretrained("shears-llama-7b-50-base")
 
 instruction = "Please choose the correct answer to the question: A cactus stem is used to store\n\nAnswer1: fruit "
         "Answer2: liquid Answer3: food Answer4: spines\n\nAnswer format: answer1/answer2/answer3/answer4"