Phi-4-onnx-cpu-int4 Unofficial version

Note: This is unoffical version,just for test and dev.

This is a Phi-4 version of ONNX CPU, based on Olive https://github.com/microsoft/olive. Convert with the following command

1. Install the SDK

pip install olive-ai

pip install transformers==4.44.2

2. Convert CPU ONNX Support

olive auto-opt --model_name_or_path Your Phi-4 location --output_path Your onnx ouput location --device cpu --provider CPUExecutionProvider --precision int4 --use_model_builder --log_level 1

This is a conversion, but no specific optimization has been done. Please look forward to the official version.

Sample - Inference ONNX

import onnxruntime_genai as og
import numpy as np
import os


model_folder = "Your Phi-4-onnx-cpu-int4 location"


model = og.Model(model_folder)


tokenizer = og.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()


search_options = {}
search_options['max_length'] = 2048
search_options['past_present_share_buffer'] = False


chat_template = "<|user|>\n{input}</s>\n<|assistant|>"


text = """I have $20,000 in my savings account, where I receive a 4% profit per year and payments twice a year. Can you please tell me how long it will take for me to become a millionaire? Also, can you please explain the math step by step as if you were explaining it to an uneducated person?"""


prompt = f'{chat_template.format(input=text)}'


input_tokens = tokenizer.encode(prompt)


params = og.GeneratorParams(model)


params.set_search_options(**search_options)
params.input_ids = input_tokens


generator = og.Generator(model, params)


while not generator.is_done():
      generator.compute_logits()
      generator.generate_next_token()

      new_token = generator.get_next_tokens()[0]
      print(tokenizer_stream.decode(new_token), end='', flush=True)