gpt2-medium-indonesian / replace_token_script.py

Commit History

update <|endoftext|> tokenizer id from 50257 to 50256
34c19b7

alvin commited on

refactor tokenizer related files with eos token
08d39dc

alvin commited on