tokenizer-arena / requirements.txt
xu-song's picture
add more tokenizer
5425d5d
raw
history blame
63 Bytes
transformers
sentencepiece
tiktoken
icetk
torch
zhon
nltk
boto3