metadata

language:
  - en
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:200000
  - loss:MultipleNegativesRankingLoss
  - loss:ContrastiveLoss
base_model: BAAI/bge-base-en-v1.5
widget:
  - source_sentence: >-
      What is the best sushi restaurant in Los Angeles, aside from Urasawa which
      is impractical for regular visits?
    sentences:
      - How do I stop feeling sorry for ignorant and arrogant people?
      - What are the best sushi restaurants in Los Angeles?
      - Why do people flirt on Quora?
  - source_sentence: Why are many Quora writers lonely and/ or unemployed?
    sentences:
      - Are writers on Quora mostly lonely or have no job (unemployed)?
      - >-
        What are the attributes of monkeys belongs to Japanese-macaque monkey
        Family?
      - >-
        I want to change the education system in India. How can I have such
        power?
  - source_sentence: What is the best, and painless way to kill myself?
    sentences:
      - >-
        What is a way to commit suicide and not damaging your organs so that
        they can be donated?
      - How do I beat insomnia?
      - What is the most painless way to commit suicide?
  - source_sentence: What are ETF'S and what is the difference between ETF'S and mutual funds?
    sentences:
      - What is the difference between ETF and mutual funds?
      - What's better, an index ETF or an index mutual fund?
      - 'Income Tax: How to check pan card status?'
  - source_sentence: For what reasons can't the Olympics be held in India?
    sentences:
      - What are the best hotels to stay in Goa?
      - When will Olympics be held in India?
      - When will India qualify for the FIFA World Cup?
datasets:
  - sentence-transformers/quora-duplicates
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy
  - cosine_accuracy_threshold
  - cosine_f1
  - cosine_f1_threshold
  - cosine_precision
  - cosine_recall
  - cosine_ap
  - cosine_mcc
  - average_precision
  - f1
  - precision
  - recall
  - threshold
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: SentenceTransformer based on BAAI/bge-base-en-v1.5
    results:
      - task:
          type: binary-classification
          name: Binary Classification
        dataset:
          name: quora duplicates
          type: quora-duplicates
        metrics:
          - type: cosine_accuracy
            value: 0.833
            name: Cosine Accuracy
          - type: cosine_accuracy_threshold
            value: 0.8065301179885864
            name: Cosine Accuracy Threshold
          - type: cosine_f1
            value: 0.7630522088353413
            name: Cosine F1
          - type: cosine_f1_threshold
            value: 0.745335042476654
            name: Cosine F1 Threshold
          - type: cosine_precision
            value: 0.6705882352941176
            name: Cosine Precision
          - type: cosine_recall
            value: 0.8850931677018633
            name: Cosine Recall
          - type: cosine_ap
            value: 0.8120519897128382
            name: Cosine Ap
          - type: cosine_mcc
            value: 0.641402259734116
            name: Cosine Mcc
      - task:
          type: paraphrase-mining
          name: Paraphrase Mining
        dataset:
          name: quora duplicates dev
          type: quora-duplicates-dev
        metrics:
          - type: average_precision
            value: 0.6286866338232051
            name: Average Precision
          - type: f1
            value: 0.6032452480296708
            name: F1
          - type: precision
            value: 0.5627297495999654
            name: Precision
          - type: recall
            value: 0.6500474596592896
            name: Recall
          - type: threshold
            value: 0.7944510877132416
            name: Threshold
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: Unknown
          type: unknown
        metrics:
          - type: cosine_accuracy@1
            value: 0.9732
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9944
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9958
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9994
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.9732
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.432
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.27652
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.14606
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8392449568046333
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9654790046130339
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9826052435636259
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9955256342023989
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9852328208350886
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.983879365079365
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9794253454223505
            name: Cosine Map@100

SentenceTransformer based on BAAI/bge-base-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the mnrl and cl datasets. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: BAAI/bge-base-en-v1.5
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Datasets:
- mnrl
- cl
Language: en

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("manestay/bge-base-en-v1.5-mnrl-cl-multi")
# Run inference
sentences = [
    "For what reasons can't the Olympics be held in India?",
    'When will Olympics be held in India?',
    'When will India qualify for the FIFA World Cup?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Binary Classification

Dataset: quora-duplicates
Evaluated with BinaryClassificationEvaluator

Metric	Value
cosine_accuracy	0.833
cosine_accuracy_threshold	0.8065
cosine_f1	0.7631
cosine_f1_threshold	0.7453
cosine_precision	0.6706
cosine_recall	0.8851
cosine_ap	0.8121
cosine_mcc	0.6414

Paraphrase Mining

Dataset: quora-duplicates-dev

Evaluated with ParaphraseMiningEvaluator with these parameters:

{'add_transitive_closure': <function ParaphraseMiningEvaluator.add_transitive_closure at 0x7f26a89802c0>, 'max_pairs': 500000, 'top_k': 100}

Metric	Value
average_precision	0.6287
f1	0.6032
precision	0.5627
recall	0.65
threshold	0.7945

Information Retrieval

Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.9732
cosine_accuracy@3	0.9944
cosine_accuracy@5	0.9958
cosine_accuracy@10	0.9994
cosine_precision@1	0.9732
cosine_precision@3	0.432
cosine_precision@5	0.2765
cosine_precision@10	0.1461
cosine_recall@1	0.8392
cosine_recall@3	0.9655
cosine_recall@5	0.9826
cosine_recall@10	0.9955
cosine_ndcg@10	0.9852
cosine_mrr@10	0.9839
cosine_map@100	0.9794

Training Details

Training Datasets

mnrl

Dataset: mnrl at 451a485
Size: 100,000 training samples
Columns: anchor, positive, and negative

Approximate statistics based on the first 1000 samples:

	anchor	positive	negative
type	string	string	string
details	min: 6 tokens mean: 13.85 tokens max: 42 tokens	min: 6 tokens mean: 13.65 tokens max: 44 tokens	min: 4 tokens mean: 14.76 tokens max: 64 tokens

Samples:

anchor	positive	negative
`Why in India do we not have one on one political debate as in USA?`	`Why cant we have a public debate between politicians in India like the one in US?`	`Can people on Quora stop India Pakistan debate? We are sick and tired seeing this everyday in bulk?`
`What is OnePlus One?`	`How is oneplus one?`	`Why is OnePlus One so good?`
`Does our mind control our emotions?`	`How do smart and successful people control their emotions?`	`How can I control my positive emotions for the people whom I love but they don't care about me?`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

cl

Dataset: cl at 451a485
Size: 100,000 training samples
Columns: sentence1, sentence2, and label
Approximate statistics based on the first 1000 samples:
sentence1 sentence2 label
type string string int
details
min: 6 tokens
mean: 15.3 tokens
max: 57 tokens

min: 6 tokens
mean: 15.66 tokens
max: 56 tokens

0: ~62.00%
1: ~38.00%

	sentence1	sentence2	label
type	string	string	int
details	min: 6 tokens mean: 15.3 tokens max: 57 tokens	min: 6 tokens mean: 15.66 tokens max: 56 tokens	0: ~62.00% 1: ~38.00%

Samples:

sentence1	sentence2	label
`What is the step by step guide to invest in share market in india?`	`What is the step by step guide to invest in share market?`	`0`
`What is the story of Kohinoor (Koh-i-Noor) Diamond?`	`What would happen if the Indian government stole the Kohinoor (Koh-i-Noor) diamond back?`	`0`
`How can I increase the speed of my internet connection while using a VPN?`	`How can Internet speed be increased by hacking through DNS?`	`0`

Loss: ContrastiveLoss with these parameters:

{
    "distance_metric": "SiameseDistanceMetric.COSINE_DISTANCE",
    "margin": 0.5,
    "size_average": true
}

Evaluation Datasets

mnrl

Dataset: mnrl at 451a485
Size: 1,000 evaluation samples
Columns: anchor, positive, and negative

Approximate statistics based on the first 1000 samples:

	anchor	positive	negative
type	string	string	string
details	min: 7 tokens mean: 13.84 tokens max: 43 tokens	min: 6 tokens mean: 13.8 tokens max: 38 tokens	min: 6 tokens mean: 14.71 tokens max: 56 tokens

Samples:

anchor	positive	negative
`Which programming language is best for developing low-end games?`	`What coding language should I learn first for making games?`	`I am entering the world of video game programming and want to know what language I should learn? Because there are so many languages I do not know which one to start with. Can you recommend a language that's easy to learn and can be used with many platforms?`
`Was it appropriate for Meryl Streep to use her Golden Globes speech to attack Donald Trump?`	`Should Meryl Streep be using her position to attack the president?`	`Why did Kelly Ann Conway say that Meryl Streep incited peoples worst feelings?`
`Where can I found excellent commercial fridges in Sydney?`	`Where can I found impressive range of commercial fridges in Sydney?`	`What is the best grocery delivery service in Sydney?`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

cl

Dataset: cl at 451a485
Size: 1,000 evaluation samples
Columns: sentence1, sentence2, and label
Approximate statistics based on the first 1000 samples:
sentence1 sentence2 label
type string string int
details
min: 5 tokens
mean: 15.59 tokens
max: 59 tokens

min: 6 tokens
mean: 15.65 tokens
max: 76 tokens

0: ~63.40%
1: ~36.60%

	sentence1	sentence2	label
type	string	string	int
details	min: 5 tokens mean: 15.59 tokens max: 59 tokens	min: 6 tokens mean: 15.65 tokens max: 76 tokens	0: ~63.40% 1: ~36.60%

Samples:

sentence1	sentence2	label
`What should I ask my friend to get from UK to India?`	`What is the process of getting a surgical residency in UK after completing MBBS from India?`	`0`
`How can I learn hacking for free?`	`How can I learn to hack seriously?`	`1`
`Which is the best website to learn programming language C++?`	`Which is the best website to learn C++ Programming language for free?`	`0`

Loss: ContrastiveLoss with these parameters:

{
    "distance_metric": "SiameseDistanceMetric.COSINE_DISTANCE",
    "margin": 0.5,
    "size_average": true
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 400
per_device_eval_batch_size: 400
num_train_epochs: 100
warmup_ratio: 0.1
bf16: True
load_best_model_at_end: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 400
per_device_eval_batch_size: 400
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 100
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	mnrl loss	cl loss	quora-duplicates_cosine_ap	quora-duplicates-dev_average_precision	cosine_ndcg@10
0	0	-	-	-	0.7461	0.5988	0.9831
0.2	100	0.2804	-	-	-	-	-
0.4	200	0.2006	-	-	-	-	-
0.5	250	-	0.1153	0.0157	0.7661	0.6165	0.9839
0.6	300	0.1704	-	-	-	-	-
0.8	400	0.1459	-	-	-	-	-
1.0	500	0.1296	0.0835	0.0146	0.7860	0.6238	0.9843
1.2	600	0.1344	-	-	-	-	-
1.4	700	0.1181	-	-	-	-	-
1.5	750	-	0.0737	0.0139	0.7983	0.6263	0.9847
1.6	800	0.1176	-	-	-	-	-
1.8	900	0.119	-	-	-	-	-
2.0	1000	0.1127	0.0682	0.0133	0.8121	0.6287	0.9852

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.12.9
Sentence Transformers: 4.1.0
Transformers: 4.52.4
PyTorch: 2.7.0+cu126
Accelerate: 1.7.0
Datasets: 3.6.0
Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

ContrastiveLoss

@inproceedings{hadsell2006dimensionality,
    author={Hadsell, R. and Chopra, S. and LeCun, Y.},
    booktitle={2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06)},
    title={Dimensionality Reduction by Learning an Invariant Mapping},
    year={2006},
    volume={2},
    number={},
    pages={1735-1742},
    doi={10.1109/CVPR.2006.100}
}