Q4_K_M?
Could you make a Q4_K_M for us 24GB havers? <3
Also, please an IQ4_XS?
Q4_K_M and IQ4_XS quant files have been uploaded.
Thanks @zetasepic ! I just tried Q4_K_M and it works great. Could you also upload Q3_K_M? It’s smaller and seems like it might be better according to https://www.reddit.com/r/LocalLLaMA/comments/1fkm5vd/qwen2532bggufevaluationresults/
@zetasepic this has been fantastic. I can run the Q4_K_M on my 3090 fully on the card! Though I also second zhicwu's request for a Q3_K_M if it's not too much trouble. While I can run the Q4 with about 27tok/sec, the context length is 8k. If I try to make the context length 16K then it doesn't have enough space on the GPU and offloads dropping down to 10tok/sec. I think the Q3_K_M would be small enough to allow the 32k context length, and if that post is accurate, then I'll still have about the same quality and might even have faster inference speeds. Thank you!