AWQ 4-bit 128g version of falcon-7b-instruct!

#55
by abhinavkulkarni - opened

Hi,

I would like to draw everyone's attention to AWQ quantized version of falcon-7b-instruct model at https://huggingface.co/abhinavkulkarni/falcon-7b-instruct-w4-g64-awq.

For more on AWQ, click here.

The quantized model size on disk is 4.76GB vs 14.43GB for the original model. Similar gains could be observed for VRAM usage. The perplexity is only worse by 5%.

Please take a look and give it a try.

Thanks!

abhinavkulkarni changed discussion status to closed
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment