Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / tokenizer

2 contributors

History: 4 commits

xu-song's picture

fix unicode error: 'unicodeescape' codec can't decode bytes in position 602-608: unknown Unicode character name

bce41d0 11 months ago

tiktoken_patch.py

2.51 kB

fix unicode error: 'unicodeescape' codec can't decode bytes in position 602-608: unknown Unicode character name 11 months ago