Qwen-MNN
Collection
Qwen MNN models.
•
21 items
•
Updated
•
1
This model is a 4-bit quantized version of the MNN model exported from QwQ-32B-Preview using llmexport.
# install huggingface
pip install huggingface
# shell download
huggingface download --model 'taobao-mnn/QwQ-32B-Preview-MNN' --local_dir 'path/to/dir'
# SDK download
from huggingface_hub import snapshot_download
model_dir = snapshot_download('taobao-mnn/QwQ-32B-Preview-MNN')
# git clone
git clone https://www.modelscope.cn/taobao-mnn/QwQ-32B-Preview-MNN
# clone MNN source
git clone https://github.com/alibaba/MNN.git
# compile
cd MNN
mkdir build && cd build
cmake .. -DMNN_LOW_MEMORY=true -DMNN_CPU_WEIGHT_DEQUANT_GEMM=true -DMNN_BUILD_LLM=true -DMNN_SUPPORT_TRANSFORMER_FUSE=true
make -j
# run
./llm_demo /path/to/QwQ-32B-Preview-MNN/config.json prompt.txt