yhavinga's picture
Hack some Dutch tokenizers into it
55df72d
raw
history blame contribute delete
141 Bytes
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("yhavinga/dutch-llama-tokenizer", trust_remote_code=True)