Spaces:

xu-song
/

tokenizer-arena

Running

tokenizer-arena / utils /compress_rate_util.py

add more tokenizer

c75633b 9 months ago

83 Bytes

	"""


	中文数据：clue superclue
	英文数据：glue cnn_dailymail gigaword

	"""