Update README.md

README.md

@@ -14,7 +14,6 @@ spaces: false
 language:
 - en
 ---
-
 # Quantized Octopus V2: On-device language model for super agent
 
 This repo includes two types of quantized models: **GGUF** and **AWQ**, for our Octopus V2 model at [NexaAIDev/Octopus-v2](https://huggingface.co/NexaAIDev/Octopus-v2)
@@ -25,63 +24,112 @@ This repo includes two types of quantized models: **GGUF** and **AWQ**, for our
 
 
 # GGUF Qauntization
-Run with [Ollama](https://github.com/ollama/ollama)
 
+To run the models, please download them to your local machine using either git clone or [Hugging Face Hub](https://huggingface.co/docs/huggingface_hub/en/guides/download)
+```
+git clone https://huggingface.co/NexaAIDev/Octopus-v2-gguf-awq
+```
+
+## Run with [llama.cpp](https://github.com/ggerganov/llama.cpp) (Recommended) 
+
+1. **Clone and compile:**
+
+```bash
+git clone https://github.com/ggerganov/llama.cpp
+cd llama.cpp
+# Compile the source code:
+make
+```
+
+2. **Execute the Model:**
+
+Run the following command in the terminal:
+
+```bash
+./main -m ./path/to/octopus-v2-Q4_K_M.gguf -n 256 -p "Below is the query from the users, please call the correct function and generate the parameters to call the function.\n\nQuery: Take a selfie for me with front camera\n\nResponse:"
+```
+
+## Run with [Ollama](https://github.com/ollama/ollama)
+
+Since our models have not been uploaded to the Ollama server, please download the models and manually import them into Ollama by following these steps:
+
+1. Install Ollama on your local machine. You can also following the guide from [Ollama GitHub repository](https://github.com/ollama/ollama/blob/main/docs/import.md)
+
+```bash
+git clone https://github.com/ollama/ollama.git ollama
+```
+
+2. Locate the local Ollama directory:
+```bash
+cd ollama
+```
+
+3. Create a `Modelfile` in your directory
+```bash
+touch Modelfile
+``` 
+
+4. In the Modelfile, include a `FROM` statement with the path to your local model, and the default parameters:
+
+```bash
+FROM ./path/to/octopus-v2-Q4_K_M.gguf
+PARAMETER temperature 0
+PARAMETER num_ctx 1024
+PARAMETER stop <nexa_end>
+```
+
+5. Use the following command to add the model to Ollama:
+```bash
+ollama create octopus-v2-Q4_K_M -f Modelfile
+```
+
+6. Verify that the model has been successfully imported:
+```bash
+ollama ls
+```
+
+7. Run the mode
 ```bash
-ollama run NexaAIDev/octopus-v2-Q4_K_M
+ollama run octopus-v2-Q4_K_M "Below is the query from the users, please call the correct function and generate the parameters to call the function.\n\nQuery: Take a selfie for me with front camera\n\nResponse:"
 ```
 
 # AWQ Quantization
 Python example:
 
 ```python
+from transformers import AutoTokenizer
 from awq import AutoAWQForCausalLM
-from transformers import AutoTokenizer, GemmaForCausalLM
 import torch
 import time
 import numpy as np
-
 def inference(input_text):
-
-    tokens = tokenizer(
-        input_text,
-        return_tensors='pt'
-    ).input_ids.cuda()
-
     start_time = time.time()
+    input_ids = tokenizer(input_text, return_tensors="pt").to('cuda')
+    input_length = input_ids["input_ids"].shape[1]
     generation_output = model.generate(
-        tokens,
-        do_sample=True,
-        temperature=0.7,
-        top_p=0.95,
-        top_k=40,
-        max_new_tokens=512
+        input_ids["input_ids"],
+        do_sample=False,
+        max_length=1024
     )
     end_time = time.time()
-
-    res = tokenizer.decode(generation_output[0])
-    res = res.split(input_text)
+    # Decode only the generated part
+    generated_sequence = generation_output[:, input_length:].tolist()
+    res = tokenizer.decode(generated_sequence[0])
     latency = end_time - start_time
-    output_tokens = tokenizer.encode(res)
-    num_output_tokens = len(output_tokens)
+    num_output_tokens = len(generated_sequence[0])
     throughput = num_output_tokens / latency
-
-    return {"output": res[-1], "latency": latency, "throughput": throughput}
-
-
-model_id = "path/to/Octopus-v2-AWQ"
+    return {"output": res, "latency": latency, "throughput": throughput}
+# Initialize tokenizer and model
+model_id = "/path/to/Octopus-v2-AWQ-NexaAIDev"
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
 model = AutoAWQForCausalLM.from_quantized(model_id, fuse_layers=True,
                                           trust_remote_code=False, safetensors=True)
-tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
-
 prompts = ["Below is the query from the users, please call the correct function and generate the parameters to call the function.\n\nQuery: Can you take a photo using the back camera and save it to the default location? \n\nResponse:"]
-
 avg_throughput = []
 for prompt in prompts:
     out = inference(prompt)
     avg_throughput.append(out["throughput"])
     print("nexa model result:\n", out["output"])
-
 print("avg throughput:", np.mean(avg_throughput))
 ```
 
@@ -114,5 +162,4 @@ _Quantized with llama.cpp_
 
 
 **Acknowledgement**:  
-We sincerely thank our community members, [Mingyuan](https://huggingface.co/ThunderBeee), [Zoey](https://huggingface.co/ZY6), [Brian](https://huggingface.co/JoyboyBrian), [Perry](https://huggingface.co/PerryCheng614), [Qi](https://huggingface.co/qiqiWav), [David](https://huggingface.co/Davidqian123) for their extraordinary contributions to this quantization effort.
-
+We sincerely thank our community members, [Mingyuan](https://huggingface.co/ThunderBeee), [Zoey](https://huggingface.co/ZY6), [Brian](https://huggingface.co/JoyboyBrian), [Perry](https://huggingface.co/PerryCheng614), [Qi](https://huggingface.co/qiqiWav), [David](https://huggingface.co/Davidqian123) for their extraordinary contributions to this quantization effort.