Hugging Face's logo

Spaces:
xu-song
/
tokenizer-arena
Running

App Files Community
1
tokenizer-arena / vocab /aya_101 /__init__.py
xu-song's picture
xu-song
add aya
44c3329
raw
history blame
106 Bytes
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("CohereForAI/aya-101")