ts-bpe-24k / special_tokens_map.json
cyrilzhang's picture
Upload tokenizer
94fd2b5
raw
history blame
55 Bytes
{
"eos_token": "<|eos|>",
"pad_token": "<|pad|>"
}