xu-song's picture
update
751936e
|
raw
history blame
240 Bytes
```
added vocab (size: 54634) with 22 dummy tokens (new size: 54656)
Vocab size: 54634
训练数据
```
https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neox_japanese/tokenization_gpt_neox_japanese.py