RuntimeError: probability tensor contains either inf, nan or element < 0
I meet this issue "RuntimeError: probability tensor contains either inf, nan or element < 0"，my gpu is V100. Is there anyone no why?
I ran into this problem with 7B model. I have solved it by using bf16 precision and replacing model.half() with model.bfloat16(). Maybe you can try this.
aha，V100 doesn't support bf16. It is said that fp16 usually is ok to do inference.But i also found
It seems that codellama is pretrained with bf16
haha, I found this solution
I ran into this problem with 7B model. I have solved it by using bf16 precision and replacing
model.half()withmodel.bfloat16(). Maybe you can try this.
using llama keyword which is the foundation model 🤣
maybe you can try fp32 or 8/4bit
set load_8_bit or load_4_bit =True, it's ok. But set torch_dtype=torch.float32 is still NG. It's very strange
Maybe this is just Wizardcode issue, i try another codellama version ok