README.md

---
language:
- en
library_name: transformers
license: other
license_name: microsoft-terms-of-use
license_link: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/resolve/main/LICENSE
tags:
- text-generation-inference
- phi3
- awq
- microsoft
extra_gated_heading: Access Microsoft on Hugging Face
extra_gated_prompt: >-
  To access Phi-3 on Hugging Face, you’re required to review and agree to
  Microsoft usage license. To do this, please ensure you’re logged-in to Hugging
  Face and click below. Requests are processed immediately.
extra_gated_button_content: Acknowledge license
---

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1kFznlPlWYOrcgd7Q1NI2tYMLH_vTRuys?usp=sharing) 

# phi-3-mini-4k-instruct-awq-4bit


phi-3-mini-4k-instruct-awq-4bit is a version of the [Microsoft](https://huggingface.co/microsoft) [Phi 3 mini 4k Instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) model that was quantized using the AWQ method developed by [Lin et al. (2023)](https://arxiv.org/abs/2306.00978).

Please refer to the [Original Phi 3 mini model card](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) for details about the model preparation and training processes.

## Dependencies
- [`autoawq==0.2.5`](https://pypi.org/project/autoawq/0.2.5/) – [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) was used to quantize the phi-3 model.
- [`vllm==0.4.2`](https://pypi.org/project/vllm/0.4.2/) – [vLLM](https://github.com/vllm-project/vllm) was used to host models for benchmarking.