mistral-community

/

Mixtral-8x22B-v0.1-AWQ

like 34

Follow

Unofficial Mistral Community 188

This PR copies the fixed mistral-common v3 tokenizer from this PR, minus the chat template.

Hi @Rocketknight1
Have you tested this change yourself? Will it work without any issue?

Hi @MaziyarPanahi , it was tested against the Mistral v3 tokenizer for chat completions and the string outputs and token sequences were identical. I'm not sure if the Mistral tokenizer even allows standard string tokenizations though - is there a way to check that?

Thanks @Rocketknight1
I will merge and we can test it. It is simpler this way.

Thanks again for your contribution