Model not loading, even with 4-bit quantization
#65
by
soumodeep-semut
- opened
Tried running this with TGI, didn't work. When loading using AutoModelForCausalLM, it is giving keyerror!!
Python 3.9.16 (main, Dec 2023, 11:55:50)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> import torch
>>> model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
>>> model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ubuntu/venv/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
File "/home/ubuntu/venv/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 1064, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
File "/home/ubuntu/venv/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 761, in __getitem__
raise KeyError(key)
KeyError: 'mixtral'
Hi
@soumodeep-semut
, pip install -U transformers
should fix your issue