ocisd4
/

llama_tokenizer_ext_zhtw

Model card Files Files and versions Community

JohnCheng commited on Jan 15

Commit

a017e9a

•

1 Parent(s): 52abbf9

Update README.md

Files changed (1) hide show

README.md +1 -2

README.md CHANGED Viewed

@@ -5,8 +5,7 @@ tokenizer = LlamaTokenizer.from_pretrained(
         'ocisd4/llama_tokenizer_ext_zhtw',
         pad_token='<unk>',
         add_bos_token=True,
-        add_eos_token=False,
-        use_auth_token=True,
 )
 #vocab size: 36128

         'ocisd4/llama_tokenizer_ext_zhtw',
         pad_token='<unk>',
         add_bos_token=True,
+        add_eos_token=False
 )
 #vocab size: 36128