README.md

metadata

license: mit
widget:
  - text: >-
      привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]супер, вот только
      проснулся, у тебя как?
    example_title: Dialog example 1
  - text: привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]норм
    example_title: Dialog example 2
  - text: привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]норм, у тя как?
    example_title: Dialog example 3

This classification model is based on cointegrated/rubert-tiny2. The model should be used to produce relevance and specificity of the last message in the context of a dialogue.

The labels explanation:

• relevance: is the last message in the dialogue relevant in the context of the full dialogue
• specificity: is the last message in the dialogue interesting and promotes the continuation of the dialogue

The preferable length of the dialogue is 4 where the last message is needed to be estimated

It is pretrained on corpus of dialog data and finetuned on tinkoff-ai/context_similarity. The performance of the model on validation split tinkoff-ai/context_similarity (with the best thresholds for validation samples):

	threshold	f0.5	ROC AUC
relevance	0.51	0.82	0.74
specificity	0.54	0.81	0.8

The preferable usage:

# pip install transformers
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("/mnt/chatbot_models2/chit-chat/experiments/crossencoder_hf/rubert-base-sentence/dialogs_whole")
model = AutoModelForSequenceClassification.from_pretrained("/mnt/chatbot_models2/chit-chat/experiments/crossencoder_hf/rubert-base-sentence/dialogs_whole")
# model.cuda()
inputs = tokenizer('привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]норм',
                   padding=True, max_length=128, truncation=True, add_special_tokens=False, return_tensors='pt')
with torch.inference_mode():
    logits = model(**inputs).logits
    probas = torch.sigmoid(logits)[0].cpu().detach().numpy()
print(probas)