System requirements
How much video memory is required to run the model?
what configuration is recommended to run?
P.S.
Thank you very much for your work!
It'll largely depend on what size you want to run and how fast
If you have more than a combined 50 gb of RAM/VRAM (so a 3090 + 32GB of RAM) you should be able to run Q4_K_M, not super fast though
The less you have available (or the faster you want to run it) the lower the quant level you should use
Q2_K will run pretty nicely if given 24GB of VRAM and 8-16GB of RAM
It'll largely depend on what size you want to run and how fast
If you have more than a combined 50 gb of RAM/VRAM (so a 3090 + 32GB of RAM) you should be able to run Q4_K_M, not super fast though
The less you have available (or the faster you want to run it) the lower the quant level you should use
Q2_K will run pretty nicely if given 24GB of VRAM and 8-16GB of RAM
Thank you 🙏🙏🙏