tokenizer.json里面没有中文字符

by rootsule - opened 5 days ago

5 days ago

大佬，中文分词全是[UNK]啊

neavo

Owner 5 days ago

•

模型采用的是字节级别的词表，所以 tokenizer.json 里面不是明文的中文字符，但是确实是覆盖了中文字符的
可以提供一下你的测试代码

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment