Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / tokenizer /tiktoken_patch.py

Commit History

add compress rate

814ee6b

xu-song commited on Apr 22

update

f02dd94

xu-song commited on Mar 4

fix PyO3PanicException

2461705

xu-song commited on Mar 4

fix unicode error: 'unicodeescape' codec can't decode bytes in position 602-608: unknown Unicode character name

bce41d0

xu-song commited on Mar 4

fix fastchat_t5_3b

c766a08

xu-song commited on Mar 4

fix tiktoken special tokens

adcfb97

xu-song commited on Mar 1

fix tiktoken

a6c67ec

xu-song commited on Feb 25