Could not load model error
I am getting following error :
Could not load model databricks/dolly-v2-12b with any of the following classes: (<class
'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.auto.modeling_tf_auto.TFAutoModelForCausalLM'>,
<class 'transformers.models.gpt_neox.modeling_gpt_neox.GPTNeoXForCausalLM'>)
If you're on a Mac, many models like this won't work, I'm pretty sure. See https://github.com/databrickslabs/dolly/issues/60
I am on Ubuntu
They just updated their documentation and sample code.
New sample code is working fine at my end now :
Working code :
import torch
from transformers import pipeline
generate_text = pipeline(model="databricks/dolly-v2-12b", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto")
Great! Happy the updated code fixed your issue.
does not work for me. I am on ubuntu. how is it working for others?
not working for me as well
what exact problem are you facing?
When i use device_map='auto' code works fine when i use cuda , it gives error below, i am on windows 10 with rtx2080 and cuda tool kit is already installed.
import torch
from transformers import pipeline
import time
use dolly-v2-12b if you're using Colab Pro+, using pythia-2.8b for Free Colab
generate_text = pipeline(
model="databricks/dolly-v2-2-8b",
torch_dtype=torch.bfloat16,
trust_remote_code=True,
device_map="cuda"
)
def prompt(s):
start = time.time()
response = generate_text(s)
end = time.time()
print(end - start)
return response
print(prompt("First man on the moon?"))
The argument trust_remote_code
is to be used with Auto classes. It has no effect here and is ignored.
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ Traceback (most recent call last) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ c:\Users\Ehsan\Desktop\opengpt4alltest1.py:77 in โ
โ โ
โ 74 from transformers import pipeline โ
โ 75 import time โ
โ 76 # use dolly-v2-12b if you're using Colab Pro+, using pythia-2.8b for Free Colab โ
โ โฑ 77 generate_text = pipeline( โ
โ 78 โ model="databricks/dolly-v2-2-8b", โ
โ 79 โ torch_dtype=torch.bfloat16, โ
โ 80 โ trust_remote_code=True, โ
โ โ
โ C:\Users\Ehsan\AppData\Local\Programs\Python\Python39\lib\site-packages\transformers\pipelines_ โ
โ init_.py:779 in pipeline โ
โ โ
โ 776 โ # Forced if framework already defined, inferred if it's None โ
โ 777 โ # Will load the correct model if possible โ
โ 778 โ model_classes = {"tf": targeted_task["tf"], "pt": targeted_task["pt"]} โ
โ โฑ 779 โ framework, model = infer_framework_load_model( โ
โ 780 โ โ model, โ
โ 781 โ โ model_classes=model_classes, โ
โ 782 โ โ config=config, โ
โ โ
โ C:\Users\Ehsan\AppData\Local\Programs\Python\Python39\lib\site-packages\transformers\pipelines\b โ
โ ase.py:271 in infer_framework_load_model โ
โ โ
โ 268 โ โ โ โ continue โ
โ 269 โ โ โ
โ 270 โ โ if isinstance(model, str): โ
โ โฑ 271 โ โ โ raise ValueError(f"Could not load model {model} with any of the following cl โ
โ 272 โ โ
โ 273 โ framework = "tf" if "keras.engine.training.Model" in str(inspect.getmro(model.__clas โ
โ 274 โ return framework, model โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
ValueError: Could not load model databricks/dolly-v2-2-8b with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>,
<class 'transformers.models.auto.modeling_tf_auto.TFAutoModelForCausalLM'>, <class 'transformers.models.gpt_neox.modeling_gpt_neox.GPTNeoXForCausalLM'>).
There are several things wrong here - old model version string and device_map="cuda" doesn't quite make sense. Please start over from the given example in the model card. This is also a different discussion now.
not sure if there was a solution for it yet?