what is the right tokenizer should I use for llama 3.1 8B?
#19
by
calebl
- opened
what is the right tokenizer should I use for llama 3.1 8B?
use meta-llama/Meta-Llama-3.1-8B-Instruct, tested on my machine.
it seems that the apply_chat_template for the tokenizer is weird, it seems to add on extra assistant response indicator
it seems that the apply_chat_template for the tokenizer is weird, it seems to add on extra assistant response indicator
I also meet such an issue. Have you fixed this bug?