安装llama.cpp

保存gguf文件路径：../Mixtral-8x7B-instruction-zh_V0.1.Q4_K_S.gguf

linux系统下的测试：

进入安装的llama.cpp根目录下，

在命令行界面CLI下执行：

CUDA_VISIBLE_DEVICES=0 ./main -m ../Mixtral-8x7B-instruction-zh_V0.1.Q4_K_S.gguf -n 2048 -p "今年是2024年，大后年是哪年？"

GGUF

Model size

46.7B params

Architecture

llama

4-bit

Inference Providers NEW

This model is not currently available via any of the supported Inference Providers.

The model cannot be deployed to the HF Inference API: The model has no pipeline_tag.