Hugging Face's logo

Skylaude
/
WizardLM-2-4x7B-MoE-exl2-6_0bpw

Text Generation
Transformers
Safetensors
mixtral
MoE
Merge
mergekit
Mistral
Microsoft/WizardLM-2-7B
text-generation-inference
Inference Endpoints
6-bit
exl2
Model card Files Community

WizardLM-2-4x7B-MoE-exl2-6_0bpw

This is a quantized version of WizardLM-2-4x7B-MoE an experimental MoE model made with Mergekit. Quantization was done using version 0.0.18 of ExLlamaV2.

Please be sure to set experts per token to 4 for the best results! Context length should be the same as Mistral-7B-Instruct-v0.1 (8k tokens). For instruction templates, Vicuna-v1.1 is recommended.

For more information see the original repository.

Downloads last month
5
Inference Examples
Text Generation
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.