Missing Tokenizer.json

#5
by tjake - opened

I manually used the one from non-chat model but should be here too

Hi πŸ‘‹@tjake, all the files for our base model are here: https://huggingface.co/01-ai/Yi-Coder-1.5B/tree/main

Yes that's what I used but my project depends on the tokenizer.json being checked in with each model.

Is there a reason you left it out? Would you mind adding it to the chat models?

Hi πŸ‘‹ Hello, the base model includes tokenizer.json for debugging convenience or special adjustments, while the chat model may only need tokenizer.model to improve loading and running efficiency. However, all 1.5B tokenizer.json files are the same. If your project needs it, you can download the 1.5B base tokenizer.json and add it to your project.

tjake changed discussion status to closed
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment