Spaces:

xu-song
/

tokenizer-arena

Running

add more tokenizers

f4973d4 11 months ago

106 Bytes


	from vocab.baichuan import tokenizer

	id1 = tokenizer.encode("<pad>")
	token1 = tokenizer.decode(125696)