vocab/glm_chinese/README.md

来源：

• https://github.com/THUDM/GLM/tree/main/chinese_sentencepiece
• https://huggingface.co/THUDM/glm-10b-chinese/

HF

tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-10b", trust_remote_code=True)

分词器

tokenizer_config.json

    "AutoTokenizer": [
      "tokenization_glm.GLMChineseTokenizer",
      null
      ]

其中 GLMChineseTokenizer

https://huggingface.co/THUDM/glm-10b-chinese/blob/main/tokenization_glm.py

词典

来自