Ominai / tokenizer.json
Erfan11's picture
Upload tokenizer.json with huggingface_hub
82f6159 verified
raw
history blame contribute delete
212 Bytes
{"version": "1.0", "tokens": ["[PAD]", "[CLS]", "[SEP]", "[MASK]", "[UNK]"], "special_tokens_map": {"cls_token": "[CLS]", "pad_token": "[PAD]", "sep_token": "[SEP]", "unk_token": "[UNK]", "mask_token": "[MASK]"}}