---
license: mit
---
# 🔥 MoE-Mixtral-7B-8Expert
[mixtral-8x7b](https://huggingface.co/someone13574/mixtral-8x7b-32kseqlen) is a Mixture-of-Expert (MoE) model. 
[LLaMA2-Accessory](https://github.com/Alpha-VLLM/LLaMA2-Accessory) has supported its inference and finetuning. 

## 🚀 Features
With LLaMA2-Accessory, mixtral-8x7b enjoys the following features:
1. Distributed MoE (namely instantiating experts on multiple processes/gpus)
2. Load Balancing Loss
3. Tensor Parallel and FSDP for efficiently training
4. Distributed and/or quantized inference

## 🔥 Online Demo
We host a web demo at <https://5e1109637f49baae47.gradio.live/>, which shows a mixtral-8x7b model finetuned on 
[evol-codealpaca-v1](https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1) and 
[ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k), with LoRA and Bias tuning. 
Please note that this is a temporary link, and we will update our official permanent link today.

## 💡 Tutorial
A detailed tutorial is available at <https://llama2-accessory.readthedocs.io/en/latest/projects/mixtral-8x7b.html#>