Spaces:

xu-song
/

tokenizer-arena

Running

add more tokenizer

293bad6 11 months ago

127 Bytes

	from transformers import AutoTokenizer

	tokenizer = AutoTokenizer.from_pretrained("paust/pko-t5-large", trust_remote_code=True)