metadata

base_model: BAAI/bge-base-en-v1.5
datasets: []
language:
  - en
library_name: sentence-transformers
license: apache-2.0
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_ndcg@100
  - cosine_mrr@10
  - cosine_map@100
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:10000
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: >-
      Politics is about action. The German government has to take some action on
      the issue of NSA surveillance and German privacy or it will look weak.
      Interior Minister Hans-Peter Friedrich went to Washington in July but was
      accused of “returning empty-handed” and having “not moved a single step
      forward on any of the key points”. [1] The stonewalling by the United
      States provides an opportunity for opponents to Damage Merkel’s new
      government as well as potentially to show gaps between the SDP and CSU.
      Merkel has been invited to visit Washington at some point in 2014 by
      President Obama, [2] Merkel can’t afford for her own diplomacy to have as
      little result as Friedrich’s.  [1] Deutsche Welle, ‘SPF, Greens slam
      Interior Minister Friedrich after US surveillance talks in Washington’,
      dw.de, 13 July 2013,   [2] Reuters, ‘Obama invites Merkel to visit during
      call about trade, NATO’, 8 January 2014,
    sentences:
      - what was mrs griffin accused of doing
      - are alcohol cigarettes dangerous
      - could gmo help food production
  - source_sentence: >-
      Schools such as those in the county of Harrold, TX [1] have already
      introduced laws allowing teachers to carry pistols, but largely in a
      concealed fashion. This therefore leaves children unawares and thus not
      distracted by seeing teachers prominently carrying guns. Furthermore, with
      teachers carrying concealed arms, any would-be attackers would be thrown
      by not knowing who to shoot first, which would not be the case if police
      officers were the first on the scene.  [1] McKinley, James C., ‘In Texas
      School, Teachers Carry Books and Guns’, The New York Times, 28 August
      2008,
    sentences:
      - why are teachers allowed to carry guns?
      - why is it important to prosecute
      - what is victim mentality
  - source_sentence: >-
      While any annexation would be mutually agreed there is no guarantee that
      the whole international community would see it positively; any resistance
      from groups within Lesotho and it could be a PR nightmare. Moreover the
      spin of it being a humanitarian gesture is reliant on it following through
      and improving conditions. If it succeeds then SA will likely be called
      upon to resolve other humanitarian situations in the region such as in
      Swaziland.
    sentences:
      - why is congress power so important
      - how africa is dependent on foreign aid
      - should lesotho be annexed
  - source_sentence: >-
      In the last 20 years, the number of people in the UK who identify as
      religious has declined by 20%. This shows that religion as a whole is
      becoming less important and, with it, marriage is becoming less important.
      (British Social Attitudes Survey 2007)
    sentences:
      - why is it important for people to identify as religious
      - is negotiation necessary for the government?
      - does the lawyer have to be privy to mediation
  - source_sentence: >-
      The ICC's ability to prosecute war criminals is both overstated and
      simplistic. It has no force of its own, and must rely on its member states
      to hand over criminals wanted for prosecution. This leads to cases like
      that of Serbia, where wanted war criminals like Ratko Mladic are believed
      to have been hidden with the complicity of the regime until finally handed
      over in 2011. The absence of a force or any coercive means to bring
      suspects to trial also leads to situations like that in Libya, whereby
      Colonel Gaddafi is wanted by the ICC but the prosecution's case is germane
      if he manages his grip on power. Furthermore, it relies on external
      funding to operate, and can only sustain cases so long as financial
      support exists to see them through.
    sentences:
      - does the icc prosecute war crimes
      - how to reduce phone usage
      - does evolution prove that the creator did the work
model-index:
  - name: SentenceTransformer based on BAAI/bge-base-en-v1.5
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.186
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.544
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.6685
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.7995
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.186
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.18133333333333332
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.13369999999999999
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.07995000000000001
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.186
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.544
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.6685
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.7995
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.4889853894775273
            name: Cosine Ndcg@10
          - type: cosine_ndcg@100
            value: 0.5263043331639856
            name: Cosine Ndcg@100
          - type: cosine_mrr@10
            value: 0.38976746031746196
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.39800392651408967
            name: Cosine Map@100

SentenceTransformer based on BAAI/bge-base-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: BAAI/bge-base-en-v1.5
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 tokens
Similarity Function: Cosine Similarity
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("MugheesAwan11/bge-base-arguana-dataset-10k-2k-e1")
# Run inference
sentences = [
    "The ICC's ability to prosecute war criminals is both overstated and simplistic. It has no force of its own, and must rely on its member states to hand over criminals wanted for prosecution. This leads to cases like that of Serbia, where wanted war criminals like Ratko Mladic are believed to have been hidden with the complicity of the regime until finally handed over in 2011. The absence of a force or any coercive means to bring suspects to trial also leads to situations like that in Libya, whereby Colonel Gaddafi is wanted by the ICC but the prosecution's case is germane if he manages his grip on power. Furthermore, it relies on external funding to operate, and can only sustain cases so long as financial support exists to see them through.",
    'does the icc prosecute war crimes',
    'does evolution prove that the creator did the work',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Dataset: dim_768
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.186
cosine_accuracy@3	0.544
cosine_accuracy@5	0.6685
cosine_accuracy@10	0.7995
cosine_precision@1	0.186
cosine_precision@3	0.1813
cosine_precision@5	0.1337
cosine_precision@10	0.08
cosine_recall@1	0.186
cosine_recall@3	0.544
cosine_recall@5	0.6685
cosine_recall@10	0.7995
cosine_ndcg@10	0.489
cosine_ndcg@100	0.5263
cosine_mrr@10	0.3898
cosine_map@100	0.398

Training Details

Training Dataset

Unnamed Dataset

Size: 10,000 training samples
Columns: positive and anchor
Approximate statistics based on the first 1000 samples:
positive anchor
type string string
details
min: 29 tokens
mean: 203.36 tokens
max: 512 tokens

min: 4 tokens
mean: 9.5 tokens
max: 25 tokens

	positive	anchor
type	string	string
details	min: 29 tokens mean: 203.36 tokens max: 512 tokens	min: 4 tokens mean: 9.5 tokens max: 25 tokens

Samples:

positive	anchor
The act of killing is emotionally damaging To actually be involved in the death of another person is an incredibly traumatic experience. Soldiers coming back from war often suffer from ‘post-traumatic stress disorder’ which suggests that being in a situation in which you have to take another persons life has a long lasting impact on your mental health. This is also true for people who are not directly involved in the act of killing. For instance, the people who worked on developing the atomic bomb described an incredible guilt for what they had created even though they were not involved in the decision to drop the bombs. The same traumatic experiences would likely affect the person responsible for pulling the lever.	`what is a killing and how can it affect the brain?`
Deal with Corruption Guinea-Bissau’s institutions have become too corrupt to deal with the drug problem and require support. The police, army and judiciary have all been implicated in the drug trade. The involvement of state officials in drug trafficking means that criminals are not prosecuted against. When two soldiers and a civilian were apprehended with 635kg (worth £25.4 million in 2013), they were detained and then immediately released with Colonel Arsenio Blade claiming ‘They were on the road hitching a ride’1. Judges are often bribed or sent death threats when faced with sentencing those involved in the drug trade. The USA has provided restructuring assistance to institutions which have reduced corruption, such as in the Mexico Merida Initiative, and could do the same with Guinea Bissau. 1) Vulliamy,E. ‘How a tiny West African country became the world’s first narco state’, The Guardian, 9 March 2008 2) Corcoran,P. ‘Mexico Judicial Reforms Go Easy On Corrupt Judges’, In Sight Crime, 16 February 2012	`what has changed guinea bissau`
Western countries already benefit from extremely liberal laws. The USA is at present far better than most countries in their respect and regard for civil liberties. New security measures do not greatly compromise this liberty, and the US measures are at the very least comparable with similar measures already in effect in other democratic developed countries, e.g. Spain and the UK, which have had to cope with domestic terrorism for far longer than the USA. The facts speak for themselves – the USA enjoys a healthy western-liberalism the likes of which most of the world’s people cannot even conceive of. The issue of the erosion of a few minor liberties of (states like the US’s) citizens should be overlooked in favour of the much greater issue of protecting the very existence of that state. [1] [1] Zetter, Kim, ‘The Patriot Act Is Your Friend’, Wired, 24 February 2004, , accessed 9 September 2011	`which political philosophy is true about the usa?`

Loss: MatryoshkaLoss with these parameters:

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768
    ],
    "matryoshka_weights": [
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_train_batch_size: 32
per_device_eval_batch_size: 16
learning_rate: 2e-05
num_train_epochs: 1
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: True
tf32: True
load_best_model_at_end: True
optim: adamw_torch_fused
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 32
per_device_eval_batch_size: 16
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: True
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_map@100
0.0319	10	0.5613	-
0.0639	20	0.4543	-
0.0958	30	0.2893	-
0.1278	40	0.2127	-
0.1597	50	0.1528	-
0.1917	60	0.1689	-
0.2236	70	0.1812	-
0.2556	80	0.1531	-
0.2875	90	0.1685	-
0.3195	100	0.1666	-
0.3514	110	0.1504	-
0.3834	120	0.139	-
0.4153	130	0.1174	-
0.4473	140	0.1602	-
0.4792	150	0.178	-
0.5112	160	0.1481	-
0.5431	170	0.1145	-
0.5751	180	0.1502	-
0.6070	190	0.1189	-
0.6390	200	0.1648	-
0.6709	210	0.2004	-
0.7029	220	0.1565	-
0.7348	230	0.1447	-
0.7668	240	0.1411	-
0.7987	250	0.1326	-
0.8307	260	0.1562	-
0.8626	270	0.1571	-
0.8946	280	0.1211	-
0.9265	290	0.1399	-
0.9585	300	0.1884	-
0.9904	310	0.1537	-
1.0	313	-	0.398

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.10.14
Sentence Transformers: 3.0.1
Transformers: 4.41.2
PyTorch: 2.1.2+cu121
Accelerate: 0.31.0
Datasets: 2.19.1
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}