Why cant load to 12GB GPU?
#1
by
robert1968
- opened
Hi,
i have RTX3060 12GB GPU, but get CUDA out of memory. i was able to load and run TheBloke_samantha-1.2-mistral-7B-AWQ which is ~ similar size.
i used AutoAWQ Model loader.
any help appreciated.
thx