|
--- |
|
library_name: transformers |
|
language: |
|
- vi |
|
- en |
|
base_model: |
|
- 5CD-AI/Vintern-1B-v2 |
|
datasets: |
|
- vidore/colpali_train_set |
|
- 5CD-AI/Viet-Doc-VQA |
|
- 5CD-AI/Viet-OCR-VQA |
|
- 5CD-AI/Viet-Doc-VQA-II |
|
tags: |
|
- colpali |
|
--- |
|
<div align="center"> |
|
<img src="colvintern.png" width="400"/> |
|
</div> |
|
|
|
## ColVintern-1B-v1 🇻🇳 ❄️ - Colpali version for Vietnamese. |
|
|
|
**What's new in ColVintern-1B-v1!** |
|
- We coded and successfully trained the **Colpali pipeline** for **Vintern-1B-v2**. The model supports RAG by extracting embedding vectors for questions and images containing related information. |
|
- This is the first experimental version, trained on the [**Colpali dataset**](https://huggingface.co/datasets/vidore/colpali_train_set) for English and **5%** of the image-based question-answer pairs we have for Vietnamese. |
|
- The model achieves results nearly equivalent to Colpali version 1, with strong support for Vietnamese texts and only 1 billion parameters compared to current 2B-3B Colpali models. |
|
|
|
## Colpali Benchmarks |
|
|
|
We tested on the [**ViDoRe benchmark**](https://huggingface.co/collections/vidore/vidore-benchmark-667173f98e70a1c0fa4db00d) from the Colpali paper. The **TabF** and **Shift** test datasets were not used because they are in French. We plan to expand to multiple languages in the near future. |
|
|
|
| | ArxivQ | DocQ | InfoQ | TATQ | AI | Energy | Gov. | Health. | Avg. | |
|
|:------------------------------:|:--------:|:------:|:-------:|:------:|:------:|:--------:|:-------:|:---------:|:--------:| |
|
| **Unstructured** Text only | | | | | | | | | | |
|
| - BM25 | - | 34.1 | - | 44.0 | 90.4 | 78.3 | 78.8 | 82.6 | - | |
|
| - BGE-M3 | - | 28.4 | - | 36.1 | 88.4 | 76.8 | 77.7 | 84.6 | - | |
|
| **Unstructured** + OCR | | | | | | | | | | |
|
| - BM25 | 31.6 | 36.8 | 62.9 | 62.7 | 92.8 | 85.9 | 83.9 | 87.2 | 68.0 | |
|
| - BGE-M3 | 31.4 | 25.7 | 60.1 | 50.5 | 90.2 | 83.6 | 84.9 | 91.1 | 64.7 | |
|
| **Unstructured** + Captioning | | | | | | | | | | |
|
| - BM25 | 40.1 | 38.4 | 70.0 | 61.5 | 88.0 | 84.7 | 82.7 | 89.2 | 69.3 | |
|
| - BGE-M3 | 35.7 | 32.9 | 71.9 | 43.8 | 88.8 | 83.3 | 80.4 | 91.3 | 66.0 | |
|
| **Contrastive VLMs** | | | | | | | | | | |
|
| - Jina-CLIP | 25.4 | 11.9 | 35.5 | 3.3 | 15.2 | 19.7 | 21.4 | 20.8 | 19.2 | |
|
| - Nomic-vision | 17.1 | 10.7 | 30.1 | 2.7 | 12.9 | 10.9 | 11.4 | 15.7 | 13.9 | |
|
| - SigLIP (Vanilla) | 43.2 | 30.3 | 64.1 | 26.2 | 62.5 | 65.7 | 66.1 | 79.1 | 54.7 | |
|
| **Colpali** | | | | | | | | | | |
|
| - SigLIP (Vanilla) | 43.2 | 30.3 | 64.1 | 26.2 | 62.5 | 65.7 | 66.1 | 79.1 | 54.7 | |
|
| - BiSigLIP (+fine-tuning) | 58.5 | 32.9 | 70.5 | 30.5 | 74.3 | 73.7 | 74.2 | 82.3 | 62.1 | |
|
| - BiPali (+LLM) | 56.5 | 30.0 | 67.4 | 33.4 | 71.2 | 61.9 | 73.8 | 73.6 | 58.5 | |
|
| - ColPali (+Late Inter.) | **79.1** | **54.4** | 81.8 | **65.8** | **96.2** | **91.0** | **92.7** | 94.4 | **81.3** | |
|
| **Ours** | | | | | | | | | | |
|
| - ColVintern-1B (+Late Inter.) | 71.6 | 48.3 | **84.6** | 59.6 | 92.9 | 88.7 | 89.4 | **95.2** | 78.8 | |
|
|
|
We are expanding the training dataset for upcoming versions, including adding hard negative mining techniques, increasing GPU VRAM, etc., to achieve better results. |
|
|
|
## Examples |
|
|
|
Input Images: |
|
<div style="display: flex; gap: 20px;"> |
|
<img src="ex1.jpg" width="300"/> |
|
<img src="ex2.jpg" width="300"/> |
|
</div> |
|
|
|
Input Queries: |
|
``` |
|
queries = ["Cảng Hải Phòng thông báo gì ?","Phí giao hàng bao nhiêu ?"] |
|
``` |
|
|
|
Output Scores: |
|
| Query | Image 1 Score | Image 2 Score | |
|
|--------------------------------------|---------------|---------------| |
|
| Chuyện gì xảy ra với quốc lộ 5 TP Hải Phòng ? | 62.4333 | 59.9523 | |
|
| Phí giao hàng bao nhiêu ? | 60.7748 | 62.8654 | |
|
|
|
|
|
|
|
## Quickstart: |
|
|
|
Colab: https://colab.research.google.com/drive/1-y8HLiyS0oCj7Vpy4i7FsJ1A6kU7ROca?usp=sharing |
|
|
|
```python |
|
import torch |
|
from PIL import Image |
|
from transformers import AutoModel, AutoTokenizer, AutoProcessor |
|
import matplotlib.pyplot as plt |
|
|
|
model_name = "5CD-AI/ColVintern-1B-v1" |
|
|
|
processor = AutoProcessor.from_pretrained( |
|
model_name, |
|
trust_remote_code=True |
|
) |
|
model = AutoModel.from_pretrained( |
|
model_name, |
|
torch_dtype=torch.bfloat16, |
|
low_cpu_mem_usage=True, |
|
trust_remote_code=True, |
|
).eval().cuda() |
|
|
|
#!wget https://huggingface.co/5CD-AI/ColVintern-1B-v1/resolve/main/ex1.jpg |
|
#!wget https://huggingface.co/5CD-AI/ColVintern-1B-v1/resolve/main/ex2.jpg |
|
|
|
images = [Image.open("ex1.jpg"),Image.open("ex2.jpg")] |
|
batch_images = processor.process_images(images) |
|
|
|
queries = [ |
|
"Cảng Hải Phòng thông báo gì ?", |
|
"Phí giao hàng bao nhiêu ?", |
|
] |
|
|
|
batch_queries = processor.process_queries(queries) |
|
|
|
batch_images["pixel_values"] = batch_images["pixel_values"].cuda().bfloat16() |
|
batch_images["input_ids"] = batch_images["input_ids"].cuda() |
|
batch_images["attention_mask"] = batch_images["attention_mask"].cuda().bfloat16() |
|
batch_queries["input_ids"] = batch_queries["input_ids"].cuda() |
|
batch_queries["attention_mask"] = batch_queries["attention_mask"].cuda().bfloat16() |
|
|
|
with torch.no_grad(): |
|
image_embeddings = model(**batch_images) |
|
query_embeddings = model(**batch_queries) |
|
|
|
scores = processor.score_multi_vector(query_embeddings, image_embeddings) |
|
|
|
max_scores, max_indices = torch.max(scores, dim=1) |
|
# In ra kết quả cho mỗi câu hỏi |
|
for i, query in enumerate(queries): |
|
print(f"Câu hỏi: '{query}'") |
|
print(f"Điểm số: {max_scores[i].item()}\n") |
|
plt.figure(figsize=(5,5)) |
|
plt.imshow(images[max_indices[i]]) |
|
plt.show() |
|
``` |
|
|
|
## Citation |
|
|
|
``` |
|
|
|
``` |
