| # AWQ 4bit Inference | |
| We integrated [AWQ](https://github.com/mit-han-lab/llm-awq) into FastChat to provide **efficient and accurate** 4bit LLM inference. | |
| ## Install AWQ | |
| Setup environment (please refer to [this link](https://github.com/mit-han-lab/llm-awq#install) for more details): | |
| ```bash | |
| conda create -n fastchat-awq python=3.10 -y | |
| conda activate fastchat-awq | |
| # cd /path/to/FastChat | |
| pip install --upgrade pip # enable PEP 660 support | |
| pip install -e . # install fastchat | |
| git clone https://github.com/mit-han-lab/llm-awq repositories/llm-awq | |
| cd repositories/llm-awq | |
| pip install -e . # install awq package | |
| cd awq/kernels | |
| python setup.py install # install awq CUDA kernels | |
| ``` | |
| ## Chat with the CLI | |
| ```bash | |
| # Download quantized model from huggingface | |
| # Make sure you have git-lfs installed (https://git-lfs.com) | |
| git lfs install | |
| git clone https://huggingface.co/mit-han-lab/vicuna-7b-v1.3-4bit-g128-awq | |
| # You can specify which quantized model to use by setting --awq-ckpt | |
| python3 -m fastchat.serve.cli \ | |
| --model-path models/vicuna-7b-v1.3-4bit-g128-awq \ | |
| --awq-wbits 4 \ | |
| --awq-groupsize 128 | |
| ``` | |
| ## Benchmark | |
| * Through **4-bit weight quantization**, AWQ helps to run larger language models within the device memory restriction and prominently accelerates token generation. All benchmarks are done with group_size 128. | |
| * Benchmark on NVIDIA RTX A6000: | |
| | Model | Bits | Max Memory (MiB) | Speed (ms/token) | AWQ Speedup | | |
| | --------------- | ---- | ---------------- | ---------------- | ----------- | | |
| | vicuna-7b | 16 | 13543 | 26.06 | / | | |
| | vicuna-7b | 4 | 5547 | 12.43 | 2.1x | | |
| | llama2-7b-chat | 16 | 13543 | 27.14 | / | | |
| | llama2-7b-chat | 4 | 5547 | 12.44 | 2.2x | | |
| | vicuna-13b | 16 | 25647 | 44.91 | / | | |
| | vicuna-13b | 4 | 9355 | 17.30 | 2.6x | | |
| | llama2-13b-chat | 16 | 25647 | 47.28 | / | | |
| | llama2-13b-chat | 4 | 9355 | 20.28 | 2.3x | | |
| * NVIDIA RTX 4090: | |
| | Model | AWQ 4bit Speed (ms/token) | FP16 Speed (ms/token) | AWQ Speedup | | |
| | --------------- | ------------------------- | --------------------- | ----------- | | |
| | vicuna-7b | 8.61 | 19.09 | 2.2x | | |
| | llama2-7b-chat | 8.66 | 19.97 | 2.3x | | |
| | vicuna-13b | 12.17 | OOM | / | | |
| | llama2-13b-chat | 13.54 | OOM | / | | |
| * NVIDIA Jetson Orin: | |
| | Model | AWQ 4bit Speed (ms/token) | FP16 Speed (ms/token) | AWQ Speedup | | |
| | --------------- | ------------------------- | --------------------- | ----------- | | |
| | vicuna-7b | 65.34 | 93.12 | 1.4x | | |
| | llama2-7b-chat | 75.11 | 104.71 | 1.4x | | |
| | vicuna-13b | 115.40 | OOM | / | | |
| | llama2-13b-chat | 136.81 | OOM | / | | |