how to load this?

#1
by Adenialzz - opened

how to load this? any code snippet for reference?

@Adenialzz


import requests
import PIL
import timm
import torch

device = "cuda"
model_name = "resnet50_clip.openai"

# Load the model
model = timm.create_model(model_name, pretrained=True)
model.eval()
model.to(device)

# Load the processor
data_config = timm.data.resolve_model_data_config(model)
processor = timm.data.create_transform(**data_config, is_training=False)

# Example Image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = PIL.Image.open(requests.get(url, stream=True).raw)

with torch.no_grad():
    input_tensor = processor(image).unsqueeze(0).to(device) # Shape 1 x 3 x 224 x 224
    output_tensor = model(input_tensor) # Shape 1 x 1024


PyTorch Image Models org

@Adenialzz I missed getting the model cards up on these, but yeah @pablomm answer is 100% ... these are the image tower weights of OpenAI ResNet CLIP models without any classifier head, so they're good for features or fine-tuning but if you want the full image+text CLIP model, you'll want to be looking at the transformers CLIP model or the OpenCLIP equivalent...

May i know if I can load the above files from local? something like after download resnet50_clip_openai locally.

model = timm.create_model(
model_name,
pretrained=True,
pretrained_cfg={'file': 'path_to_file/resnet50_clip_openai'})

PyTorch Image Models org

@forwardcjj yes, that will work but path should be to the safetensors or pth file not the folder

PyTorch Image Models org

though someday will support folder and look for config files, etc

Exception has occurred: TypeError
unsupported operand type(s) for *: 'NoneType' and 'NoneType'
File "/home/ubuntu/resnet50_clip_openai/load_model.py", line 12, in
model = timm.create_model(model_name, pretrained=True, pretrained_cfg={'file': '/home/ubuntu/resnet50_clip_openai/open_clip_model.safetensors'})
TypeError: unsupported operand type(s) for *: 'NoneType' and 'NoneType'

PyTorch Image Models org
edited Sep 19

@forwardcjj oops, use pretrained_cfg_overlay instead of pretrained_cfg ... the file key needs to be merged with the existing pretrained_cfg (not overwrite completely) as some values are needed from the original...

model = timm.create_model('resnete50_clip.openai', pretrained=True, pretrained_cfg_overlay={'file': '/home/ubuntu/resnet50_clip_openai/open_clip_model.safetensors'})

thanks. @rwightman . working on my side now. I may bother u more since I need to extract pretrained resnet50 vision model into my own network. I appreciate your time.

Sign up or log in to comment