TheBloke

/

Mistral-7B-Instruct-v0.1-GPTQ

like 78

Getting This error while using specific GPU
( eg [input_ids = tokenizer(prompt_template, return_tensors='pt').to('cuda:3'])
but when using multiple GPU it is Working Fine [input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()]

ERROR:
Traceback (most recent call last):
File "/home2/asgtestdrive2023/Text2SQL/testvenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 266, in getattr
return self.data[item]
KeyError: 'shape'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home2/asgtestdrive2023/Text2SQL/testvenv/lib/python3.10/site-packages/flask/app.py", line 1455, in wsgi_app
response = self.full_dispatch_request()
File "/home2/asgtestdrive2023/Text2SQL/testvenv/lib/python3.10/site-packages/flask/app.py", line 869, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home2/asgtestdrive2023/Text2SQL/testvenv/lib/python3.10/site-packages/flask/app.py", line 867, in full_dispatch_request
rv = self.dispatch_request()
File "/home2/asgtestdrive2023/Text2SQL/testvenv/lib/python3.10/site-packages/flask/app.py", line 852, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/home2/asgtestdrive2023/Text2SQL/sql_api.py", line 16, in chat
return jsonify({'response' : get_data(query)})
File "/home2/asgtestdrive2023/Text2SQL/sql_api.py", line 20, in get_data
return text2sql(input)
File "/home2/asgtestdrive2023/Text2SQL/pipeline_v3.py", line 205, in text2sql
output = model.generate(inputs=input_ids, temperature=0.3, do_sample=True, top_p=0.95, top_k=1, max_new_tokens=512)
File "/home2/asgtestdrive2023/Text2SQL/testvenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home2/asgtestdrive2023/Text2SQL/testvenv/lib/python3.10/site-packages/transformers/generation/utils.py", line 1459, in generate
batch_size = inputs_tensor.shape[0]
File "/home2/asgtestdrive2023/Text2SQL/testvenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 268, in getattr
raise AttributeError
AttributeError