TheBloke

/

StableBeluga2-70B-GPTQ

like 90

│ /opt/miniconda3/envs/ptcu117/lib/python3.10/site-packages/transformers/models/llama/modeling_lla │
│ ma.py:197 in forward │
│ │
│ 194 │ │ bsz, q_len, _ = hidden_states.size() │
│ 195 │ │ │
│ 196 │ │ query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self. │
│ ❱ 197 │ │ key_states = self.k_proj(hidden_states).view(bsz, q_len, self.num_heads, self.he │
│ 198 │ │ value_states = self.v_proj(hidden_states).view(bsz, q_len, self.num_heads, self. │
│ 199 │ │ │
│ 200 │ │ kv_seq_len = key_states.shape[-2] │
│ │
│ /opt/miniconda3/envs/ptcu117/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /opt/miniconda3/envs/ptcu117/lib/python3.10/site-packages/auto_gptq/nn_modules/qlinear_old.py:24 │
│ 9 in forward │
│ │
│ 246 │ │ │ weight = weight.reshape(weight.shape[0] * weight.shape[1], weight.shape[2]) │
│ 247 │ │ │ │
│ 248 │ │ │ out = torch.matmul(x.half(), weight) │
│ ❱ 249 │ │ out = out.half().reshape(out_shape) │
│ 250 │ │ out = out + self.bias if self.bias is not None else out │
│ 251 │ │ return out │
│ 252 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: shape '[1, 613, 8192]' is invalid for input of size 627712

1 * 613 * 8192 / 627712 = 8, looks like the size of K projection vector is not right.

My code is exactly copied from model card.