abu1111/bge-synthesisQA-query-encoder

Model Description

We introduce bge-synthesisQA, a retriever specifically designed for the conversational QA scenario. It can handle conversational query which combine dialogue history with the current query. It is built on top of the bge retriever. Please note that bge-synthesisQA is a dual encoder consisting of a query encoder and a context encoder. This repository is only for the query encoder of bge-synthesisQA for getting the query embeddings, and you also need the context encoder to get context embeddings, which can be found here. Both query encoder and context encoder share the same tokenizer.

Other Resources

Benchmark Results

	Average			Doc2Dial			QuAC			QReCC
	top-1	top-5	top-20	top-1	top-5	top-20	top-1	top-5	top-20	top-1	top-5	top-20
bge-large-en-v1.5	37.75	71.55	91.54	32.55	66.13	86.98	47.93	74.26	92.02	32.76	74.27	95.63
bge-synthesisQA	51.72	83.12	95.02	38.72	71.39	88.86	50.71	83.57	97.13	65.73	94.41	99.07

Retrieval results across five multi-turn QA datasets (Doc2Dial, QuAC, QReCC) with the average top-1 top-5 and top-20 recall scores.

How to use

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('abu1111/bge-synthesisQA-query-encoder')
query_encoder = AutoModel.from_pretrained('abu1111/bge-synthesisQA-query-encoder')
context_encoder = AutoModel.from_pretrained('abu1111/bge-synthesisQA-context-encoder')

query = [
    {"role": "user", "content": "Can you help me understand how to choose a health insurance plan?"},
    {"role": "agent", "content": "Sure, are you looking for individual coverage or a family plan?"},
    {"role": "user", "content": "I need a family plan."}
]
contexts = [
    "Health Insurance Guide | Choosing the Right Plan \nWhen selecting a health insurance plan, it's important to consider your family's specific needs. Factors such as the number of family members, medical history, and budget should all play a role in your decision. Plans are typically divided into categories: Bronze, Silver, Gold, and Platinum, with each offering different levels of coverage and premiums. For families with frequent medical visits, a plan with a higher premium but lower out-of-pocket costs (like a Gold or Platinum plan) might make sense. Conversely, if your family is generally healthy, a Bronze plan with lower premiums might be more cost-effective.",
    "Health Insurance Basics | Individual vs. Family Coverage \nIndividual health insurance plans cover only one person, while family plans provide coverage for an entire household. Family plans usually have a higher premium but include shared deductibles and out-of-pocket maximums. When choosing a family plan, consider factors like the total coverage provided, the network of doctors and hospitals, and whether the plan includes benefits such as pediatric care. Understanding these distinctions can help you make an informed decision about the best option for you and your loved ones."
]

## convert query into a format as follows:
## user: {user}\nagent: {agent}\nuser: {user}
formatted_query = '\n'.join([turn['role'] + ": " + turn['content'] for turn in query]).strip()

## get query and context embeddings
query_input = tokenizer(formatted_query, return_tensors='pt')
ctx_input = tokenizer(contexts, padding=True, truncation=True, max_length=512, return_tensors='pt')
query_emb = query_encoder(**query_input).last_hidden_state[:, 0, :]
ctx_emb = context_encoder(**ctx_input).last_hidden_state[:, 0, :]

## Compute similarity scores using dot product
similarities = query_emb.matmul(ctx_emb.transpose(0, 1)) # (1, num_ctx)

## rank the similarity (from highest to lowest)
ranked_results = torch.argsort(similarities, dim=-1, descending=True) # (1, num_ctx)

Evaluations on Multi-Turn QA Retrieval Benchmark

(UPDATE!!) We evaluate multi-turn QA retrieval on three datasets: Doc2Dial, QuAC, and QReCC, which can be found in the ChatRAG Bench. The evaluation scripts can be found here.

License

bge-synthesisQA is built on top of bge. We refer users to the original license of the BAAI/bge-large-en-v1.5 model. bge-synthesisQA is also subject to the Terms of Use.