How much RAM required?
#2
by
BBLL3456
- opened
I have 32GB RAM and 16GB GPU and can't seem to load the model, always get killed. May I know what is the specs requirement to run this for inference? Thanks.
Maybe you can try to specify torch_dtype=torch.float16 or torch_dtype=torch.bfloat16 to load the model. In default situation transformers will load the model in fp32 and use about 30GB RAM, which is too much for 32GB.