convert model to safetensors
This is the state_dict of the model converted to safetensors.
I think. It is a whole lot smaller than ckpt.pt, but that might be due to the fact that it doesn't include all the optimizer data?
Hello, have you tried using this directly in huggingface transformers? If so, which type of huggingface transformers model should I use that with? And is there any special config I need to use?
I've been trying to get it to work with regular clip, but have been running into errors.
something like this
import numpy.core.multiarray
torch.serialization.add_safe_globals([
(numpy.core.multiarray.scalar, 'numpy.core.multiarray.scalar'),
numpy.dtype,
numpy.dtypes.Float64DType
])
ckpt = torch.load('ckpt.pt', 'cpu', weights_only=True)
state_dict = ckpt['state_dict']
import safetensors.torch
safetensors.torch.save_file(state_dict, Path('model.safetensors'), {'format': 'pt'})
and I'm not sure whether it can work with transformers alone. I think it would be CLIPTextModelWithProjection, but the keys in the state dict are named different things than the models that's usually used with, so I don't know if it can be run without the authors' code https://github.com/ivonajdenkoska/tulip
Ah okay yeah, cuz I ran into naming issues when trying to use the model directly as a huggingface clip