Update tokenizer_config.json

#4
by basujindal - opened

Configuring bos, eos, pad and unk tokens helps load the tokenizer successfully, otherwise getting a recursion error "RecursionError: maximum recursion depth exceeded while getting the str of an object." As done here: https://huggingface.co/medalpaca/medalpaca-13b/discussions/7

kbressem changed pull request status to merged
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment