xu-song's picture
add more tokenizer
293bad6
raw
history blame
134 Bytes
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("lmsys/fastchat-t5-3b-v1.0", trust_remote_code=True)