|
--- |
|
language: |
|
- fr |
|
- it |
|
- de |
|
- es |
|
- en |
|
license: apache-2.0 |
|
inference: |
|
parameters: |
|
temperature: 0.5 |
|
widget: |
|
- messages: |
|
- role: user |
|
content: What is your favorite condiment? |
|
datasets: |
|
- HuggingFaceH4/ultrachat_200k |
|
- liuhaotian/LLaVA-Instruct-150K |
|
--- |
|
|
|
|
|
## Use below code to develop |
|
|
|
```py |
|
# pip install -U accelerate transformers torch |
|
|
|
import torch |
|
from transformers import pipeline |
|
pipe = pipeline("text-generation", model="vicky4s4s/Mixtral-instruct-56B", torch_dtype=torch.bfloat16, device_map="auto") |
|
|
|
messages = [ |
|
|
|
{"role": "user", "content": "what is your name?"}, |
|
] |
|
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
|
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) |
|
print(outputs[0]["generated_text"]) |
|
|
|
``` |
|
# Model Card for Mixtral-8x7B |
|
|
|
The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts. The Mixtral-8x7B outperforms Llama 2 70B on most benchmarks we tested. |
|
|
|
For full details of this model please read our [release blog post](https://mistral.ai/news/mixtral-of-experts/). |
|
|
|
## Warning |
|
This repo contains weights that are compatible with [vLLM](https://github.com/vllm-project/vllm) serving of the model as well as Hugging Face [transformers](https://github.com/huggingface/transformers) library. It is based on the original Mixtral [torrent release](magnet:?xt=urn:btih:5546272da9065eddeb6fcd7ffddeef5b75be79a7&dn=mixtral-8x7b-32kseqlen&tr=udp%3A%2F%http://2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=http%3A%2F%http://2Ftracker.openbittorrent.com%3A80%2Fannounce), but the file format and parameter names are different. Please note that model cannot (yet) be instantiated with HF. |
|
|
|
## Instruction format |
|
|
|
This format must be strictly respected, otherwise the model will generate sub-optimal outputs. |
|
|
|
The template used to build a prompt for the Instruct model is defined as follows: |
|
``` |
|
<s> [INST] Instruction [/INST] Model answer</s> [INST] Follow-up instruction [/INST] |
|
``` |
|
Note that `<s>` and `</s>` are special tokens for beginning of string (BOS) and end of string (EOS) while [INST] and [/INST] are regular strings. |
|
|
|
|
|
## Limitations |
|
|
|
The Mixtral-8x7B Instruct model is a quick demonstration that the base model can be easily fine-tuned to achieve compelling performance. |
|
It does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to |
|
make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs. |
|
|
|
## Created by: |
|
|
|
Vignesh, [email protected] |
