Update README.md
Browse files
README.md
CHANGED
@@ -186,35 +186,12 @@ python -m executorch.examples.models.llama.export_llama \
|
|
186 |
--metadata '{"get_bos_id":199999, "get_eos_ids":[200020,199999]}' \
|
187 |
--output_name="phi4-mini-8dq4w.pte"
|
188 |
```
|
189 |
-
|
190 |
-
## Run model with pybindings
|
191 |
-
```
|
192 |
-
export TOKENIZER="/path/to/tokenizer.json"
|
193 |
-
export TOKENIZER_CONFIG="/path/to/tokenizer_config.json"
|
194 |
-
export PROMPT="<|system|><|end|><|user|>Hey, are you conscious? Can you talk to me?<|end|><|assistant|>"
|
195 |
-
python -m executorch.examples.models.llama.runner.native \
|
196 |
-
--model phi_4_mini \
|
197 |
-
--pte phi4-mini-8dq4w.pte \
|
198 |
-
-kv \
|
199 |
-
--tokenizer ${TOKENIZER} \
|
200 |
-
--tokenizer_config ${TOKENIZER_CONFIG} \
|
201 |
-
--prompt "${PROMPT}" \
|
202 |
-
--params "${PARAMS}" \
|
203 |
-
--max_len 128 \
|
204 |
-
--temperature 0
|
205 |
-
```
|
206 |
-
|
207 |
-
The output is:
|
208 |
-
|
209 |
-
```
|
210 |
-
Hello! I am Phi, an AI developed by Microsoft. I am not conscious in the way humans are, but I am here to help and converse with you. How can I assist you today?Hello! I am Phi, an AI developed by Microsoft. I am not conscious in the way humans are, but I am here to help and converse with you. How can I assist you today?Hello! I am Phi, an AI developed by Microsoft. I am not conscious in the way humans are, but I am here to
|
211 |
-
```
|
212 |
-
|
213 |
-
Note: the runner does not currently recongize the stop token from Phi 4 Mini, so it generates text beyond when it should stop.
|
214 |
|
215 |
## Running in a mobile app
|
216 |
-
The model can be run in a mobile app. See [instructions](https://pytorch.org/executorch/main/llm/llama-demo-ios.html) for doing this
|
217 |
-
|
|
|
|
|
218 |
|
219 |
# Disclaimer
|
220 |
PyTorch has not performed safety evaluations or red teamed the quantized models. Performance characteristics, outputs, and behaviors may differ from the original models. Users are solely responsible for selecting appropriate use cases, evaluating and mitigating for accuracy, safety, and fairness, ensuring security, and complying with all applicable laws and regulations.
|
|
|
186 |
--metadata '{"get_bos_id":199999, "get_eos_ids":[200020,199999]}' \
|
187 |
--output_name="phi4-mini-8dq4w.pte"
|
188 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
189 |
|
190 |
## Running in a mobile app
|
191 |
+
The model can be run in a mobile app. See [instructions](https://pytorch.org/executorch/main/llm/llama-demo-ios.html) for doing this in iOS.
|
192 |
+
On iPhone 15 Pro, the model runs at 17.3 tokens/sec and uses 3206 Mb of memory.
|
193 |
+
|
194 |
+

|
195 |
|
196 |
# Disclaimer
|
197 |
PyTorch has not performed safety evaluations or red teamed the quantized models. Performance characteristics, outputs, and behaviors may differ from the original models. Users are solely responsible for selecting appropriate use cases, evaluating and mitigating for accuracy, safety, and fairness, ensuring security, and complying with all applicable laws and regulations.
|