|
--- |
|
license: mit |
|
--- |
|
|
|
# **Phi-3.5-mini-instruct-onnx-gpu Unofficial version** |
|
|
|
<b><span style="text-decoration:underline">Note: This is unoffical version,just for test and dev.</span></b> |
|
|
|
This is a Phi-3.5-mini-instruct version of ONNX GPU, based on ONNX Runtime for GenAI [https://github.com/microsoft/onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai). Convert with the following command |
|
|
|
## **1. Install the SDK** |
|
|
|
|
|
``` |
|
|
|
pip install torch transformers onnx onnxruntime |
|
|
|
|
|
pip install --pre onnxruntime-genai |
|
|
|
``` |
|
|
|
## **2. Convert GPU ONNX Support** |
|
|
|
|
|
```bash |
|
|
|
python3 -m onnxruntime_genai.models.builder -m microsoft/Phi-3.5-mini-instruct -o ./onnx-gpu -p int4 -e cuda -c ./Phi-3.5-mini-instruct |
|
|
|
``` |
|
|
|
|
|
|
|
This is a conversion, but no specific optimization has been done. Please look forward to the official version. |
