Steelskull

/

L3.3-MS-Nevoria-70b

like 27

I use the quants from bartowski without any issues.

You might want to try higher-quantized versions like Q4_K_S

You might want to try higher-quantized versions like Q4_K_S

I would like to, but I cant. Using 2_K is already redlining my pc, I have tried IQ2_XS and that works.

I am using https://huggingface.co/mradermacher/L3.3-MS-Nevoria-70b-GGUF/tree/main 2_k and https://huggingface.co/mradermacher/L3.3-MS-Nevoria-70b-i1-GGUF/tree/main 2_k_S, so I am not 100% sure of this is the right place to post this but both produce unusable garbage.

Hey there I think i saw your post on discord as well

have you tried redownloading the GGUF? there is a chance, although rarely, where the model gets corrupted during download. what template are you running as well?

I think it's a low quant issue, as he had a similar issue here:
https://huggingface.co/SicariusSicariiStuff/Negative_LLAMA_70B/discussions/4

Using 2 different models pretty much confirms that very low quants of ~2 bpw just don't cut it.
Pretty sure Q3 wouldn't have any issues.

I'll be closing this as problem solved unless something new happens

thanks for the help @SicariusSicariiStuff !