Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / tokenizer

2 contributors

History: 2 commits

xu-song's picture

fix tiktoken special tokens

adcfb97 8 months ago

tiktoken_patch.py

2.15 kB

fix tiktoken special tokens 8 months ago