How this model was created?
According to config, this model converted by autoawq with zero_point=False.
However, using this option fails conversion on autoawq==0.2.7.post3 and torch==2.3.1+cu121.
What is the versions of autoawq and pytorch you used to create this quantized model?
https://huggingface.co/jakiAJK/DeepSeek-R1-Distill-Qwen-7B_AWQ/blob/main/config.json#L32
Hey
@bilzard
, Big Fan of you from Kaggle(Eedi competition).
In Eedi competition, team at No.2 spot mentioned they used Intel AutoRound (https://github.com/intel/auto-round) to quantise models, and I noticed they support AWQ and GPTQ conversion as well, so 1 stone, 2 birds.
I use Intel AutoRound to quantise models. auto-gptq==0.7.1 auto-round==0.4.4 autoawq==0.2.8 torch==2.5.1+cu124
These are parameters I used, tweaking based on loss per layer. If final loss was more than 0.6, I restart by increasing iters and nsamples
Params - nsamples=320, iters=350, seqlen=2000