Add new SentenceTransformer model.

d21f609 verified 5 months ago

25.6 kB

	---
	base_model: BAAI/bge-base-en-v1.5
	datasets: []
	language:
	- en
	library_name: sentence-transformers
	license: apache-2.0
	metrics:
	- cosine_accuracy@1
	- cosine_accuracy@3
	- cosine_accuracy@5
	- cosine_accuracy@10
	- cosine_precision@1
	- cosine_precision@3
	- cosine_precision@5
	- cosine_precision@10
	- cosine_recall@1
	- cosine_recall@3
	- cosine_recall@5
	- cosine_recall@10
	- cosine_ndcg@10
	- cosine_ndcg@100
	- cosine_mrr@10
	- cosine_map@100
	pipeline_tag: sentence-similarity
	tags:
	- sentence-transformers
	- sentence-similarity
	- feature-extraction
	- generated_from_trainer
	- dataset_size:10000
	- loss:MatryoshkaLoss
	- loss:MultipleNegativesRankingLoss
	widget:
	- source_sentence: Politics is about action. The German government has to take some
	action on the issue of NSA surveillance and German privacy or it will look weak.
	Interior Minister Hans-Peter Friedrich went to Washington in July but was accused
	of “returning empty-handed” and having “not moved a single step forward on any
	of the key points”. [1] The stonewalling by the United States provides an opportunity
	for opponents to Damage Merkel’s new government as well as potentially to show
	gaps between the SDP and CSU. Merkel has been invited to visit Washington at some
	point in 2014 by President Obama, [2] Merkel can’t afford for her own diplomacy
	to have as little result as Friedrich’s. [1] Deutsche Welle, ‘SPF, Greens slam
	Interior Minister Friedrich after US surveillance talks in Washington’, dw.de,
	13 July 2013, [2] Reuters, ‘Obama invites Merkel to visit during call about
	trade, NATO’, 8 January 2014,
	sentences:
	- what was mrs griffin accused of doing
	- are alcohol cigarettes dangerous
	- could gmo help food production
	- source_sentence: Schools such as those in the county of Harrold, TX [1] have already
	introduced laws allowing teachers to carry pistols, but largely in a concealed
	fashion. This therefore leaves children unawares and thus not distracted by seeing
	teachers prominently carrying guns. Furthermore, with teachers carrying concealed
	arms, any would-be attackers would be thrown by not knowing who to shoot first,
	which would not be the case if police officers were the first on the scene. [1]
	McKinley, James C., ‘In Texas School, Teachers Carry Books and Guns’, The New
	York Times, 28 August 2008,
	sentences:
	- why are teachers allowed to carry guns?
	- why is it important to prosecute
	- what is victim mentality
	- source_sentence: While any annexation would be mutually agreed there is no guarantee
	that the whole international community would see it positively; any resistance
	from groups within Lesotho and it could be a PR nightmare. Moreover the spin of
	it being a humanitarian gesture is reliant on it following through and improving
	conditions. If it succeeds then SA will likely be called upon to resolve other
	humanitarian situations in the region such as in Swaziland.
	sentences:
	- why is congress power so important
	- how africa is dependent on foreign aid
	- should lesotho be annexed
	- source_sentence: In the last 20 years, the number of people in the UK who identify
	as religious has declined by 20%. This shows that religion as a whole is becoming
	less important and, with it, marriage is becoming less important. (British Social
	Attitudes Survey 2007)
	sentences:
	- why is it important for people to identify as religious
	- is negotiation necessary for the government?
	- does the lawyer have to be privy to mediation
	- source_sentence: The ICC's ability to prosecute war criminals is both overstated
	and simplistic. It has no force of its own, and must rely on its member states
	to hand over criminals wanted for prosecution. This leads to cases like that of
	Serbia, where wanted war criminals like Ratko Mladic are believed to have been
	hidden with the complicity of the regime until finally handed over in 2011. The
	absence of a force or any coercive means to bring suspects to trial also leads
	to situations like that in Libya, whereby Colonel Gaddafi is wanted by the ICC
	but the prosecution's case is germane if he manages his grip on power. Furthermore,
	it relies on external funding to operate, and can only sustain cases so long as
	financial support exists to see them through.
	sentences:
	- does the icc prosecute war crimes
	- how to reduce phone usage
	- does evolution prove that the creator did the work
	model-index:
	- name: SentenceTransformer based on BAAI/bge-base-en-v1.5
	results:
	- task:
	type: information-retrieval
	name: Information Retrieval
	dataset:
	name: dim 768
	type: dim_768
	metrics:
	- type: cosine_accuracy@1
	value: 0.186
	name: Cosine Accuracy@1
	- type: cosine_accuracy@3
	value: 0.544
	name: Cosine Accuracy@3
	- type: cosine_accuracy@5
	value: 0.6685
	name: Cosine Accuracy@5
	- type: cosine_accuracy@10
	value: 0.7995
	name: Cosine Accuracy@10
	- type: cosine_precision@1
	value: 0.186
	name: Cosine Precision@1
	- type: cosine_precision@3
	value: 0.18133333333333332
	name: Cosine Precision@3
	- type: cosine_precision@5
	value: 0.13369999999999999
	name: Cosine Precision@5
	- type: cosine_precision@10
	value: 0.07995000000000001
	name: Cosine Precision@10
	- type: cosine_recall@1
	value: 0.186
	name: Cosine Recall@1
	- type: cosine_recall@3
	value: 0.544
	name: Cosine Recall@3
	- type: cosine_recall@5
	value: 0.6685
	name: Cosine Recall@5
	- type: cosine_recall@10
	value: 0.7995
	name: Cosine Recall@10
	- type: cosine_ndcg@10
	value: 0.4889853894775273
	name: Cosine Ndcg@10
	- type: cosine_ndcg@100
	value: 0.5263043331639856
	name: Cosine Ndcg@100
	- type: cosine_mrr@10
	value: 0.38976746031746196
	name: Cosine Mrr@10
	- type: cosine_map@100
	value: 0.39800392651408967
	name: Cosine Map@100
	---

	# SentenceTransformer based on BAAI/bge-base-en-v1.5

	This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

	## Model Details

	### Model Description
	- Model Type: Sentence Transformer
	- Base model: [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) <!-- at revision a5beb1e3e68b9ab74eb54cfd186867f64f240e1a -->
	- Maximum Sequence Length: 512 tokens
	- Output Dimensionality: 768 tokens
	- Similarity Function: Cosine Similarity
	<!-- - Training Dataset: Unknown -->
	- Language: en
	- License: apache-2.0

	### Model Sources

	- Documentation: [Sentence Transformers Documentation](https://sbert.net)
	- Repository: [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
	- Hugging Face: [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)

	### Full Model Architecture

	```
	SentenceTransformer(
	(0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel
	(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
	(2): Normalize()
	)
	```

	## Usage

	### Direct Usage (Sentence Transformers)

	First install the Sentence Transformers library:

	```bash
	pip install -U sentence-transformers
	```

	Then you can load this model and run inference.
	```python
	from sentence_transformers import SentenceTransformer

	# Download from the 🤗 Hub
	model = SentenceTransformer("MugheesAwan11/bge-base-arguana-dataset-10k-2k-e1")
	# Run inference
	sentences = [
	"The ICC's ability to prosecute war criminals is both overstated and simplistic. It has no force of its own, and must rely on its member states to hand over criminals wanted for prosecution. This leads to cases like that of Serbia, where wanted war criminals like Ratko Mladic are believed to have been hidden with the complicity of the regime until finally handed over in 2011. The absence of a force or any coercive means to bring suspects to trial also leads to situations like that in Libya, whereby Colonel Gaddafi is wanted by the ICC but the prosecution's case is germane if he manages his grip on power. Furthermore, it relies on external funding to operate, and can only sustain cases so long as financial support exists to see them through.",
	'does the icc prosecute war crimes',
	'does evolution prove that the creator did the work',
	]
	embeddings = model.encode(sentences)
	print(embeddings.shape)
	# [3, 768]

	# Get the similarity scores for the embeddings
	similarities = model.similarity(embeddings, embeddings)
	print(similarities.shape)
	# [3, 3]
	```

	<!--
	### Direct Usage (Transformers)

	<details><summary>Click to see the direct usage in Transformers</summary>

	</details>
	-->

	<!--
	### Downstream Usage (Sentence Transformers)

	You can finetune this model on your own dataset.

	<details><summary>Click to expand</summary>

	</details>
	-->

	<!--
	### Out-of-Scope Use

	List how the model may foreseeably be misused and address what users ought not to do with the model.
	-->

	## Evaluation

	### Metrics

	#### Information Retrieval
	* Dataset: `dim_768`
	* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

	\| Metric \| Value \|
	\|:--------------------\|:----------\|
	\| cosine_accuracy@1 \| 0.186 \|
	\| cosine_accuracy@3 \| 0.544 \|
	\| cosine_accuracy@5 \| 0.6685 \|
	\| cosine_accuracy@10 \| 0.7995 \|
	\| cosine_precision@1 \| 0.186 \|
	\| cosine_precision@3 \| 0.1813 \|
	\| cosine_precision@5 \| 0.1337 \|
	\| cosine_precision@10 \| 0.08 \|
	\| cosine_recall@1 \| 0.186 \|
	\| cosine_recall@3 \| 0.544 \|
	\| cosine_recall@5 \| 0.6685 \|
	\| cosine_recall@10 \| 0.7995 \|
	\| cosine_ndcg@10 \| 0.489 \|
	\| cosine_ndcg@100 \| 0.5263 \|
	\| cosine_mrr@10 \| 0.3898 \|
	\| cosine_map@100 \| 0.398 \|

	<!--
	## Bias, Risks and Limitations

	What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.
	-->

	<!--
	### Recommendations

	What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.
	-->

	## Training Details

	### Training Dataset

	#### Unnamed Dataset


	* Size: 10,000 training samples
	* Columns: <code>positive</code> and <code>anchor</code>
	* Approximate statistics based on the first 1000 samples:
	\| \| positive \| anchor \|
	\|:--------\|:-------------------------------------------------------------------------------------\|:--------------------------------------------------------------------------------\|
	\| type \| string \| string \|
	\| details \| <ul><li>min: 29 tokens</li><li>mean: 203.36 tokens</li><li>max: 512 tokens</li></ul> \| <ul><li>min: 4 tokens</li><li>mean: 9.5 tokens</li><li>max: 25 tokens</li></ul> \|
	* Samples:
	\| positive \| anchor \|
	\|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|:----------------------------------------------------------------\|
	\| <code>The act of killing is emotionally damaging To actually be involved in the death of another person is an incredibly traumatic experience. Soldiers coming back from war often suffer from ‘post-traumatic stress disorder’ which suggests that being in a situation in which you have to take another persons life has a long lasting impact on your mental health. This is also true for people who are not directly involved in the act of killing. For instance, the people who worked on developing the atomic bomb described an incredible guilt for what they had created even though they were not involved in the decision to drop the bombs. The same traumatic experiences would likely affect the person responsible for pulling the lever.</code> \| <code>what is a killing and how can it affect the brain?</code> \|
	\| <code>Deal with Corruption Guinea-Bissau’s institutions have become too corrupt to deal with the drug problem and require support. The police, army and judiciary have all been implicated in the drug trade. The involvement of state officials in drug trafficking means that criminals are not prosecuted against. When two soldiers and a civilian were apprehended with 635kg (worth £25.4 million in 2013), they were detained and then immediately released with Colonel Arsenio Blade claiming ‘They were on the road hitching a ride’1. Judges are often bribed or sent death threats when faced with sentencing those involved in the drug trade. The USA has provided restructuring assistance to institutions which have reduced corruption, such as in the Mexico Merida Initiative, and could do the same with Guinea Bissau. 1) Vulliamy,E. ‘How a tiny West African country became the world’s first narco state’, The Guardian, 9 March 2008 2) Corcoran,P. ‘Mexico Judicial Reforms Go Easy On Corrupt Judges’, In Sight Crime, 16 February 2012</code> \| <code>what has changed guinea bissau</code> \|
	\| <code>Western countries already benefit from extremely liberal laws. The USA is at present far better than most countries in their respect and regard for civil liberties. New security measures do not greatly compromise this liberty, and the US measures are at the very least comparable with similar measures already in effect in other democratic developed countries, e.g. Spain and the UK, which have had to cope with domestic terrorism for far longer than the USA. The facts speak for themselves – the USA enjoys a healthy western-liberalism the likes of which most of the world’s people cannot even conceive of. The issue of the erosion of a few minor liberties of (states like the US’s) citizens should be overlooked in favour of the much greater issue of protecting the very existence of that state. [1] [1] Zetter, Kim, ‘The Patriot Act Is Your Friend’, Wired, 24 February 2004, , accessed 9 September 2011</code> \| <code>which political philosophy is true about the usa?</code> \|
	* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
	```json
	{
	"loss": "MultipleNegativesRankingLoss",
	"matryoshka_dims": [
	768
	],
	"matryoshka_weights": [
	1
	],
	"n_dims_per_step": -1
	}
	```

	### Training Hyperparameters
	#### Non-Default Hyperparameters

	- `eval_strategy`: epoch
	- `per_device_train_batch_size`: 32
	- `per_device_eval_batch_size`: 16
	- `learning_rate`: 2e-05
	- `num_train_epochs`: 1
	- `lr_scheduler_type`: cosine
	- `warmup_ratio`: 0.1
	- `bf16`: True
	- `tf32`: True
	- `load_best_model_at_end`: True
	- `optim`: adamw_torch_fused
	- `batch_sampler`: no_duplicates

	#### All Hyperparameters
	<details><summary>Click to expand</summary>

	- `overwrite_output_dir`: False
	- `do_predict`: False
	- `eval_strategy`: epoch
	- `prediction_loss_only`: True
	- `per_device_train_batch_size`: 32
	- `per_device_eval_batch_size`: 16
	- `per_gpu_train_batch_size`: None
	- `per_gpu_eval_batch_size`: None
	- `gradient_accumulation_steps`: 1
	- `eval_accumulation_steps`: None
	- `learning_rate`: 2e-05
	- `weight_decay`: 0.0
	- `adam_beta1`: 0.9
	- `adam_beta2`: 0.999
	- `adam_epsilon`: 1e-08
	- `max_grad_norm`: 1.0
	- `num_train_epochs`: 1
	- `max_steps`: -1
	- `lr_scheduler_type`: cosine
	- `lr_scheduler_kwargs`: {}
	- `warmup_ratio`: 0.1
	- `warmup_steps`: 0
	- `log_level`: passive
	- `log_level_replica`: warning
	- `log_on_each_node`: True
	- `logging_nan_inf_filter`: True
	- `save_safetensors`: True
	- `save_on_each_node`: False
	- `save_only_model`: False
	- `restore_callback_states_from_checkpoint`: False
	- `no_cuda`: False
	- `use_cpu`: False
	- `use_mps_device`: False
	- `seed`: 42
	- `data_seed`: None
	- `jit_mode_eval`: False
	- `use_ipex`: False
	- `bf16`: True
	- `fp16`: False
	- `fp16_opt_level`: O1
	- `half_precision_backend`: auto
	- `bf16_full_eval`: False
	- `fp16_full_eval`: False
	- `tf32`: True
	- `local_rank`: 0
	- `ddp_backend`: None
	- `tpu_num_cores`: None
	- `tpu_metrics_debug`: False
	- `debug`: []
	- `dataloader_drop_last`: False
	- `dataloader_num_workers`: 0
	- `dataloader_prefetch_factor`: None
	- `past_index`: -1
	- `disable_tqdm`: False
	- `remove_unused_columns`: True
	- `label_names`: None
	- `load_best_model_at_end`: True
	- `ignore_data_skip`: False
	- `fsdp`: []
	- `fsdp_min_num_params`: 0
	- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
	- `fsdp_transformer_layer_cls_to_wrap`: None
	- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
	- `deepspeed`: None
	- `label_smoothing_factor`: 0.0
	- `optim`: adamw_torch_fused
	- `optim_args`: None
	- `adafactor`: False
	- `group_by_length`: False
	- `length_column_name`: length
	- `ddp_find_unused_parameters`: None
	- `ddp_bucket_cap_mb`: None
	- `ddp_broadcast_buffers`: False
	- `dataloader_pin_memory`: True
	- `dataloader_persistent_workers`: False
	- `skip_memory_metrics`: True
	- `use_legacy_prediction_loop`: False
	- `push_to_hub`: False
	- `resume_from_checkpoint`: None
	- `hub_model_id`: None
	- `hub_strategy`: every_save
	- `hub_private_repo`: False
	- `hub_always_push`: False
	- `gradient_checkpointing`: False
	- `gradient_checkpointing_kwargs`: None
	- `include_inputs_for_metrics`: False
	- `eval_do_concat_batches`: True
	- `fp16_backend`: auto
	- `push_to_hub_model_id`: None
	- `push_to_hub_organization`: None
	- `mp_parameters`:
	- `auto_find_batch_size`: False
	- `full_determinism`: False
	- `torchdynamo`: None
	- `ray_scope`: last
	- `ddp_timeout`: 1800
	- `torch_compile`: False
	- `torch_compile_backend`: None
	- `torch_compile_mode`: None
	- `dispatch_batches`: None
	- `split_batches`: None
	- `include_tokens_per_second`: False
	- `include_num_input_tokens_seen`: False
	- `neftune_noise_alpha`: None
	- `optim_target_modules`: None
	- `batch_eval_metrics`: False
	- `batch_sampler`: no_duplicates
	- `multi_dataset_batch_sampler`: proportional

	</details>

	### Training Logs
	\| Epoch \| Step \| Training Loss \| dim_768_cosine_map@100 \|
	\|:-------:\|:-------:\|:-------------:\|:----------------------:\|
	\| 0.0319 \| 10 \| 0.5613 \| - \|
	\| 0.0639 \| 20 \| 0.4543 \| - \|
	\| 0.0958 \| 30 \| 0.2893 \| - \|
	\| 0.1278 \| 40 \| 0.2127 \| - \|
	\| 0.1597 \| 50 \| 0.1528 \| - \|
	\| 0.1917 \| 60 \| 0.1689 \| - \|
	\| 0.2236 \| 70 \| 0.1812 \| - \|
	\| 0.2556 \| 80 \| 0.1531 \| - \|
	\| 0.2875 \| 90 \| 0.1685 \| - \|
	\| 0.3195 \| 100 \| 0.1666 \| - \|
	\| 0.3514 \| 110 \| 0.1504 \| - \|
	\| 0.3834 \| 120 \| 0.139 \| - \|
	\| 0.4153 \| 130 \| 0.1174 \| - \|
	\| 0.4473 \| 140 \| 0.1602 \| - \|
	\| 0.4792 \| 150 \| 0.178 \| - \|
	\| 0.5112 \| 160 \| 0.1481 \| - \|
	\| 0.5431 \| 170 \| 0.1145 \| - \|
	\| 0.5751 \| 180 \| 0.1502 \| - \|
	\| 0.6070 \| 190 \| 0.1189 \| - \|
	\| 0.6390 \| 200 \| 0.1648 \| - \|
	\| 0.6709 \| 210 \| 0.2004 \| - \|
	\| 0.7029 \| 220 \| 0.1565 \| - \|
	\| 0.7348 \| 230 \| 0.1447 \| - \|
	\| 0.7668 \| 240 \| 0.1411 \| - \|
	\| 0.7987 \| 250 \| 0.1326 \| - \|
	\| 0.8307 \| 260 \| 0.1562 \| - \|
	\| 0.8626 \| 270 \| 0.1571 \| - \|
	\| 0.8946 \| 280 \| 0.1211 \| - \|
	\| 0.9265 \| 290 \| 0.1399 \| - \|
	\| 0.9585 \| 300 \| 0.1884 \| - \|
	\| 0.9904 \| 310 \| 0.1537 \| - \|
	\| 1.0 \| 313 \| - \| 0.398 \|

	* The bold row denotes the saved checkpoint.

	### Framework Versions
	- Python: 3.10.14
	- Sentence Transformers: 3.0.1
	- Transformers: 4.41.2
	- PyTorch: 2.1.2+cu121
	- Accelerate: 0.31.0
	- Datasets: 2.19.1
	- Tokenizers: 0.19.1

	## Citation

	### BibTeX

	#### Sentence Transformers
	```bibtex
	@inproceedings{reimers-2019-sentence-bert,
	title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
	author = "Reimers, Nils and Gurevych, Iryna",
	booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
	month = "11",
	year = "2019",
	publisher = "Association for Computational Linguistics",
	url = "https://arxiv.org/abs/1908.10084",
	}
	```

	#### MatryoshkaLoss
	```bibtex
	@misc{kusupati2024matryoshka,
	title={Matryoshka Representation Learning},
	author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
	year={2024},
	eprint={2205.13147},
	archivePrefix={arXiv},
	primaryClass={cs.LG}
	}
	```

	#### MultipleNegativesRankingLoss
	```bibtex
	@misc{henderson2017efficient,
	title={Efficient Natural Language Response Suggestion for Smart Reply},
	author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
	year={2017},
	eprint={1705.00652},
	archivePrefix={arXiv},
	primaryClass={cs.CL}
	}
	```

	<!--
	## Glossary

	Clearly define terms in order to be accessible across audiences.
	-->

	<!--
	## Model Card Authors

	Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.
	-->

	<!--
	## Model Card Contact

	Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.
	-->