bge-large-en-mnli-anli

This model is a fine-tuned version of BAAI/bge-large-en on the glue and ANLI dataset.

Model description

RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder. Shitao Xiao, Zheng Liu, Yingxia Shao, Zhao Cao, arXiv 2022

How to use the model

With the zero-shot classification pipeline

The model can be loaded with the zero-shot-classification pipeline like so:

from transformers import pipeline
classifier = pipeline("zero-shot-classification",
                      model="mjwong/bge-large-en-mnli-anli")

You can then use this pipeline to classify sequences into any of the class names you specify.

sequence_to_classify = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(sequence_to_classify, candidate_labels)

If more than one candidate label can be correct, pass multi_class=True to calculate each class independently:

candidate_labels = ['travel', 'cooking', 'dancing', 'exploration']
classifier(sequence_to_classify, candidate_labels, multi_class=True)

With manual PyTorch

The model can also be applied on NLI tasks like so:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
# device = "cuda:0" or "cpu"
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
model_name = "mjwong/bge-large-en-mnli-anli"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
premise = "But I thought you'd sworn off coffee."
hypothesis = "I thought that you vowed to drink more coffee."
input = tokenizer(premise, hypothesis, truncation=True, return_tensors="pt")
output = model(input["input_ids"].to(device))
prediction = torch.softmax(output["logits"][0], -1).tolist()
label_names = ["entailment", "neutral", "contradiction"]
prediction = {name: round(float(pred) * 100, 2) for pred, name in zip(prediction, label_names)}
print(prediction)

Eval results

The model was also evaluated using the dev sets for MultiNLI and test sets for ANLI. The metric used is accuracy.

Datasets	mnli_dev_m	mnli_dev_mm	anli_test_r1	anli_test_r2	anli_test_r3
bge-large-en-mnli-anli	0.846	0.842	0.602	0.451	0.452

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1

Framework versions

• Transformers 4.28.1
• Pytorch 2.0.1+cu118
• Datasets 2.11.0
• Tokenizers 0.13.3