--- base_model: BAAI/bge-base-en-v1.5 datasets: [] language: - en library_name: sentence-transformers license: apache-2.0 metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_ndcg@100 - cosine_mrr@10 - cosine_map@100 pipeline_tag: sentence-similarity tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:10000 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss widget: - source_sentence: Politics is about action. The German government has to take some action on the issue of NSA surveillance and German privacy or it will look weak. Interior Minister Hans-Peter Friedrich went to Washington in July but was accused of “returning empty-handed” and having “not moved a single step forward on any of the key points”. [1] The stonewalling by the United States provides an opportunity for opponents to Damage Merkel’s new government as well as potentially to show gaps between the SDP and CSU. Merkel has been invited to visit Washington at some point in 2014 by President Obama, [2] Merkel can’t afford for her own diplomacy to have as little result as Friedrich’s. [1] Deutsche Welle, ‘SPF, Greens slam Interior Minister Friedrich after US surveillance talks in Washington’, dw.de, 13 July 2013, [2] Reuters, ‘Obama invites Merkel to visit during call about trade, NATO’, 8 January 2014, sentences: - what was mrs griffin accused of doing - are alcohol cigarettes dangerous - could gmo help food production - source_sentence: Schools such as those in the county of Harrold, TX [1] have already introduced laws allowing teachers to carry pistols, but largely in a concealed fashion. This therefore leaves children unawares and thus not distracted by seeing teachers prominently carrying guns. Furthermore, with teachers carrying concealed arms, any would-be attackers would be thrown by not knowing who to shoot first, which would not be the case if police officers were the first on the scene. [1] McKinley, James C., ‘In Texas School, Teachers Carry Books and Guns’, The New York Times, 28 August 2008, sentences: - why are teachers allowed to carry guns? - why is it important to prosecute - what is victim mentality - source_sentence: While any annexation would be mutually agreed there is no guarantee that the whole international community would see it positively; any resistance from groups within Lesotho and it could be a PR nightmare. Moreover the spin of it being a humanitarian gesture is reliant on it following through and improving conditions. If it succeeds then SA will likely be called upon to resolve other humanitarian situations in the region such as in Swaziland. sentences: - why is congress power so important - how africa is dependent on foreign aid - should lesotho be annexed - source_sentence: In the last 20 years, the number of people in the UK who identify as religious has declined by 20%. This shows that religion as a whole is becoming less important and, with it, marriage is becoming less important. (British Social Attitudes Survey 2007) sentences: - why is it important for people to identify as religious - is negotiation necessary for the government? - does the lawyer have to be privy to mediation - source_sentence: The ICC's ability to prosecute war criminals is both overstated and simplistic. It has no force of its own, and must rely on its member states to hand over criminals wanted for prosecution. This leads to cases like that of Serbia, where wanted war criminals like Ratko Mladic are believed to have been hidden with the complicity of the regime until finally handed over in 2011. The absence of a force or any coercive means to bring suspects to trial also leads to situations like that in Libya, whereby Colonel Gaddafi is wanted by the ICC but the prosecution's case is germane if he manages his grip on power. Furthermore, it relies on external funding to operate, and can only sustain cases so long as financial support exists to see them through. sentences: - does the icc prosecute war crimes - how to reduce phone usage - does evolution prove that the creator did the work model-index: - name: SentenceTransformer based on BAAI/bge-base-en-v1.5 results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.186 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.544 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.6685 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.7995 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.186 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.18133333333333332 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.13369999999999999 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.07995000000000001 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.186 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.544 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.6685 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.7995 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.4889853894775273 name: Cosine Ndcg@10 - type: cosine_ndcg@100 value: 0.5263043331639856 name: Cosine Ndcg@100 - type: cosine_mrr@10 value: 0.38976746031746196 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.39800392651408967 name: Cosine Map@100 --- # SentenceTransformer based on BAAI/bge-base-en-v1.5 This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 tokens - **Similarity Function:** Cosine Similarity - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("MugheesAwan11/bge-base-arguana-dataset-10k-2k-e1") # Run inference sentences = [ "The ICC's ability to prosecute war criminals is both overstated and simplistic. It has no force of its own, and must rely on its member states to hand over criminals wanted for prosecution. This leads to cases like that of Serbia, where wanted war criminals like Ratko Mladic are believed to have been hidden with the complicity of the regime until finally handed over in 2011. The absence of a force or any coercive means to bring suspects to trial also leads to situations like that in Libya, whereby Colonel Gaddafi is wanted by the ICC but the prosecution's case is germane if he manages his grip on power. Furthermore, it relies on external funding to operate, and can only sustain cases so long as financial support exists to see them through.", 'does the icc prosecute war crimes', 'does evolution prove that the creator did the work', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `dim_768` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:----------| | cosine_accuracy@1 | 0.186 | | cosine_accuracy@3 | 0.544 | | cosine_accuracy@5 | 0.6685 | | cosine_accuracy@10 | 0.7995 | | cosine_precision@1 | 0.186 | | cosine_precision@3 | 0.1813 | | cosine_precision@5 | 0.1337 | | cosine_precision@10 | 0.08 | | cosine_recall@1 | 0.186 | | cosine_recall@3 | 0.544 | | cosine_recall@5 | 0.6685 | | cosine_recall@10 | 0.7995 | | cosine_ndcg@10 | 0.489 | | cosine_ndcg@100 | 0.5263 | | cosine_mrr@10 | 0.3898 | | **cosine_map@100** | **0.398** | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 10,000 training samples * Columns: positive and anchor * Approximate statistics based on the first 1000 samples: | | positive | anchor | |:--------|:-------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | positive | anchor | |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------| | The act of killing is emotionally damaging To actually be involved in the death of another person is an incredibly traumatic experience. Soldiers coming back from war often suffer from ‘post-traumatic stress disorder’ which suggests that being in a situation in which you have to take another persons life has a long lasting impact on your mental health. This is also true for people who are not directly involved in the act of killing. For instance, the people who worked on developing the atomic bomb described an incredible guilt for what they had created even though they were not involved in the decision to drop the bombs. The same traumatic experiences would likely affect the person responsible for pulling the lever. | what is a killing and how can it affect the brain? | | Deal with Corruption Guinea-Bissau’s institutions have become too corrupt to deal with the drug problem and require support. The police, army and judiciary have all been implicated in the drug trade. The involvement of state officials in drug trafficking means that criminals are not prosecuted against. When two soldiers and a civilian were apprehended with 635kg (worth £25.4 million in 2013), they were detained and then immediately released with Colonel Arsenio Blade claiming ‘They were on the road hitching a ride’1. Judges are often bribed or sent death threats when faced with sentencing those involved in the drug trade. The USA has provided restructuring assistance to institutions which have reduced corruption, such as in the Mexico Merida Initiative, and could do the same with Guinea Bissau. 1) Vulliamy,E. ‘How a tiny West African country became the world’s first narco state’, The Guardian, 9 March 2008 2) Corcoran,P. ‘Mexico Judicial Reforms Go Easy On Corrupt Judges’, In Sight Crime, 16 February 2012 | what has changed guinea bissau | | Western countries already benefit from extremely liberal laws. The USA is at present far better than most countries in their respect and regard for civil liberties. New security measures do not greatly compromise this liberty, and the US measures are at the very least comparable with similar measures already in effect in other democratic developed countries, e.g. Spain and the UK, which have had to cope with domestic terrorism for far longer than the USA. The facts speak for themselves – the USA enjoys a healthy western-liberalism the likes of which most of the world’s people cannot even conceive of. The issue of the erosion of a few minor liberties of (states like the US’s) citizens should be overlooked in favour of the much greater issue of protecting the very existence of that state. [1] [1] Zetter, Kim, ‘The Patriot Act Is Your Friend’, Wired, 24 February 2004, , accessed 9 September 2011 | which political philosophy is true about the usa? | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 768 ], "matryoshka_weights": [ 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 1 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `bf16`: True - `tf32`: True - `load_best_model_at_end`: True - `optim`: adamw_torch_fused - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 1 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: True - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | dim_768_cosine_map@100 | |:-------:|:-------:|:-------------:|:----------------------:| | 0.0319 | 10 | 0.5613 | - | | 0.0639 | 20 | 0.4543 | - | | 0.0958 | 30 | 0.2893 | - | | 0.1278 | 40 | 0.2127 | - | | 0.1597 | 50 | 0.1528 | - | | 0.1917 | 60 | 0.1689 | - | | 0.2236 | 70 | 0.1812 | - | | 0.2556 | 80 | 0.1531 | - | | 0.2875 | 90 | 0.1685 | - | | 0.3195 | 100 | 0.1666 | - | | 0.3514 | 110 | 0.1504 | - | | 0.3834 | 120 | 0.139 | - | | 0.4153 | 130 | 0.1174 | - | | 0.4473 | 140 | 0.1602 | - | | 0.4792 | 150 | 0.178 | - | | 0.5112 | 160 | 0.1481 | - | | 0.5431 | 170 | 0.1145 | - | | 0.5751 | 180 | 0.1502 | - | | 0.6070 | 190 | 0.1189 | - | | 0.6390 | 200 | 0.1648 | - | | 0.6709 | 210 | 0.2004 | - | | 0.7029 | 220 | 0.1565 | - | | 0.7348 | 230 | 0.1447 | - | | 0.7668 | 240 | 0.1411 | - | | 0.7987 | 250 | 0.1326 | - | | 0.8307 | 260 | 0.1562 | - | | 0.8626 | 270 | 0.1571 | - | | 0.8946 | 280 | 0.1211 | - | | 0.9265 | 290 | 0.1399 | - | | 0.9585 | 300 | 0.1884 | - | | 0.9904 | 310 | 0.1537 | - | | **1.0** | **313** | **-** | **0.398** | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.14 - Sentence Transformers: 3.0.1 - Transformers: 4.41.2 - PyTorch: 2.1.2+cu121 - Accelerate: 0.31.0 - Datasets: 2.19.1 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```