Hugging Face's logo

Spaces:
yhavinga
/
dutch-tokenizer-arena
Running

App Files Community
1
dutch-tokenizer-arena / vocab /falcon_7b /__init__.py
xu-song's picture
xu-song
add more tokenizer
293bad6
raw
history blame contribute delete
127 Bytes
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-7b", trust_remote_code=True)