Trouble loading in model weights at checkpoint
Hi, I am having trouble using LlavaLlamaForCausalLM to load in wisdomik/Quilt-Llava-v1.5-7b. Any idea on why this may be occurring? Should I initialize differently?
Warning I get: Some weights of the model checkpoint at wisdomik/Quilt-Llava-v1.5-7b were not used when initializing LlavaLlamaForCausalLM: ['model.vision_tower.vision_tower.vision_model.embeddings.class_embedding', 'model.vision_tower.vision_tower.vision_model.embeddings.patch_embedding.weight', 'model.vision_tower.vision_tower.vision_model.embeddings.position_embedding.weight', .........]
code line: model = LlavaLlamaForCausalLM.from_pretrained("wisdomik/Quilt-Llava-v1.5-7b", token=hft, torch_dtype=torch.float16).cuda()
Hi, Thanks for noticing this and reporting. It is a known warning of LLaVA, due to how the models are compressed or binned! see source GitHub issue here https://github.com/haotian-liu/LLaVA/issues/672
It should not affect the use of the model or the weights at all.
Thanks.