hltcoe/plaidx-large-eng-tdist-mt5xxl-engeng

ColBERT-X for English MonoLingual Retrieval using Translate-Distill

CLIR Model Setting

• Query language: English
• Query length: 32 token max
• Document language: English
• Document length: 180 token max (please use MaxP to aggregate the passage score if needed)

Model Description

Translate-Distill is a training technique that produces state-of-the-art CLIR dense retrieval model through translation and distillation. plaidx-large-eng-tdist-mt5xxl-engeng is trained with KL-Divergence from the mt5xxl MonoT5 reranker inferenced on English MS MARCO training queries and English passages. Despite using a multilingual language model as backcone, this model is trianed only on English text, which is designed for English monolingual retrieval. However, it has the ability to zero-shot to any language setup.

Teacher Models:

• t53b: castorini/monot5-3b-msmarco-10k
• mt5xxl: unicamp-dl/mt5-13b-mmarco-100k

Training Parameters

• learning rate: 5e-6
• update steps: 200,000
• nway (number of passages per query): 6 (randomly selected from 50)
• per device batch size (number of query-passage set): 8
• training GPU: 8 NVIDIA V100 with 32 GB memory

Usage

To properly load ColBERT-X models from Huggingface Hub, please use the following version of PLAID-X.

pip install PLAID-X==0.3.1

Following code snippet loads the model through Huggingface API.

from colbert.modeling.checkpoint import Checkpoint
from colbert.infra import ColBERTConfig

Checkpoint('hltcoe/plaidx-large-eng-tdist-mt5xxl-engeng', colbert_config=ColBERTConfig())

For full tutorial, please refer to the PLAID-X Jupyter Notebook, which is part of the SIGIR 2023 CLIR Tutorial.

BibTeX entry and Citation Info

Please cite the following two papers if you use the model.

@inproceedings{colbert-x,
    author = {Suraj Nair and Eugene Yang and Dawn Lawrie and Kevin Duh and Paul McNamee and Kenton Murray and James Mayfield and Douglas W. Oard},
    title = {Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models},
    booktitle = {Proceedings of the 44th European Conference on Information Retrieval (ECIR)},
    year = {2022},
    url = {https://arxiv.org/abs/2201.08471}
}

@inproceedings{translate-distill,
    author = {Eugene Yang and Dawn Lawrie and James Mayfield and Douglas W. Oard and Scott Miller},
    title = {Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation},
    booktitle = {Proceedings of the 46th European Conference on Information Retrieval (ECIR)},
    year = {2024},
    url = {https://arxiv.org/abs/2401.04810}
}