|
--- |
|
license: apache-2.0 |
|
pipeline_tag: feature-extraction |
|
tags: |
|
- clip |
|
- vision |
|
datasets: |
|
- Ziyang/yfcc15m |
|
- conceptual_captions |
|
--- |
|
<h1 align="center">UForm</h1> |
|
<h3 align="center"> |
|
Multi-Modal Inference Library<br/> |
|
For Semantic Search Applications<br/> |
|
</h3> |
|
|
|
--- |
|
|
|
UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space! |
|
|
|
This is model card of the __English only model__ with: |
|
|
|
* 4 layers BERT (2 layers for unimodal encoding and rest layers for multimodal encoding) |
|
* ViT-S/16 (image resolution is 224x224) |
|
|
|
|
|
If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual). |
|
|
|
## Evaluation |
|
|
|
The following metrics were obtained with multimodal re-ranking (text-to-image retrieval): |
|
|
|
| Dataset |Recall@1 | Recall@5 | Recall@10 | |
|
| :------ | ------: | --------: | --------: | |
|
| Zero-Shot Flickr | 0.565 | 0.790 | 0.860 | |
|
| Zero-Shot MS-COCO | 0.281 | 0.525 | 0.645 | |
|
|
|
ImageNet-Top1: 0.361 \ |
|
ImageNet-Top5: 0.608 |
|
|
|
## Installation |
|
|
|
```bash |
|
pip install uform[torch] |
|
``` |
|
|
|
## Usage |
|
|
|
To load the model: |
|
|
|
```python |
|
import uform |
|
|
|
model, processor = uform.get_model('unum-cloud/uform-vl-english-small') |
|
``` |
|
|
|
To encode data: |
|
|
|
```python |
|
from PIL import Image |
|
|
|
text = 'a small red panda in a zoo' |
|
image = Image.open('red_panda.jpg') |
|
|
|
image_data = processor.preprocess_image(image) |
|
text_data = processor.preprocess_text(text) |
|
|
|
image_features, image_embedding = model.encode_image(image_data, return_features=True) |
|
text_features, text_embedding = model.encode_text(text_data, return_features=True) |
|
joint_embedding = model.encode_multimodal(image=image_data, text=text_data) |
|
``` |
|
|
|
To get features: |
|
|
|
```python |
|
image_features, image_embedding = model.encode_image(image_data, return_features=True) |
|
text_features, text_embedding = model.encode_text(text_data, return_features=True) |
|
``` |
|
|
|
These features can later be used to produce joint multimodal encodings faster, as the first layers of the transformer can be skipped: |
|
|
|
```python |
|
joint_embedding = model.encode_multimodal( |
|
image_features=image_features, |
|
text_features=text_features, |
|
attention_mask=text_data['attention_mask'] |
|
) |
|
``` |
|
|
|
There are two options to calculate semantic compatibility between an image and a text: [Cosine Similarity](#cosine-similarity) and [Matching Score](#matching-score). |
|
|
|
### Cosine Similarity |
|
|
|
```python |
|
import torch.nn.functional as F |
|
|
|
similarity = F.cosine_similarity(image_embedding, text_embedding) |
|
``` |
|
|
|
The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match. |
|
|
|
__Pros__: |
|
|
|
- Computationally cheap. |
|
- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding. |
|
- Suitable for retrieval in large collections. |
|
|
|
__Cons__: |
|
|
|
- Takes into account only coarse-grained features. |
|
|
|
|
|
### Matching Score |
|
|
|
Unlike cosine similarity, unimodal embedding are not enough. |
|
Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match. |
|
|
|
```python |
|
score = model.get_matching_scores(joint_embedding) |
|
``` |
|
|
|
__Pros__: |
|
|
|
- Joint embedding captures fine-grained features. |
|
- Suitable for re-ranking – sorting retrieval result. |
|
|
|
__Cons__: |
|
|
|
- Resource-intensive. |
|
- Not suitable for retrieval in large collections. |
