TheBloke

/

Mixtral-8x7B-Instruct-v0.1-GPTQ

like 135

I haven't "built from source" as instructed, but the official releases of both Transformers and Auto_GPTQ are now at/beyond the dev branch suggested. Is it reasonable to assume that the latest versions have what is needed to run this?

I'm running this: GPTQ_3bit_128g_actorderTrue

On a 4090, and ultimately get this

return forward_call(*args, **kwargs)
  File "C:\Anaconda3\envs\huggingface\lib\site-packages\transformers\models\mixtral\modeling_mixtral.py", line 802, in forward
    router_logits = self.gate(hidden_states)
  File "C:\Anaconda3\envs\huggingface\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Anaconda3\envs\huggingface\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Anaconda3\envs\huggingface\lib\site-packages\auto_gptq\nn_modules\qlinear\qlinear_cuda.py", line 245, in forward
    zeros = zeros.reshape(self.scales.shape)
RuntimeError: shape '[32, 8]' is invalid for input of size 0

Making sure there are no conflicts, here are the versions from a python command line:

>>> import transformers
>>> transformers.__version__
'4.37.0'
>>> import auto_gptq
>>> auto_gptq.__version__
'0.6.0'