Tokenizer is EleutherAI/gpt-neox-20b

#4
by DanielTTY - opened

I got a tokenizer error while performing qlora with the qlora.py in their github. Installing the gpt-neox-20b tokenizer solves the problem:
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")

Sign up or log in to comment