MugheesAwan11/bge-base-citi-dataset-9k-1k-e1

SentenceTransformer based on BAAI/bge-base-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("MugheesAwan11/bge-base-citi-dataset-9k-1k-e1")
# Run inference
sentences = [
    ' be sure you had enough cash on hand to pay the fare. Channels, TTS Of course, all of that changed in the blink of an eye with the advent of ride sharing. Now getting from point A to point B is as easy as opening an app on your smart phone. Not only is it simple to book the ride, but once your account is set up, payment is absolutely seamless. No longer do you have to search your wallet for cash. The app knows who you are and the entire transaction happens in the background, without further input on the part of the rider or the driver. Payment is embedded in the experience as part of the natural flow, so you don’t have to think about it. \n 2 Treasury and Trade Solutions The invisible bank 3 This is just one example of the changes happening in today’s hyper-connected world. Artificial intelligence (AI), cloud is poised to deliver the “invisible bank,” where treasury and banking functions meld together. The continuous evolution of banking computing',
    'What is the advent of ride sharing?',
    'How is a credit card introductory APR beneficial?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.049
cosine_accuracy@3	0.115
cosine_accuracy@5	0.15
cosine_accuracy@10	0.205
cosine_precision@1	0.049
cosine_precision@3	0.0383
cosine_precision@5	0.03
cosine_precision@10	0.0205
cosine_recall@1	0.049
cosine_recall@3	0.115
cosine_recall@5	0.15
cosine_recall@10	0.205
cosine_ndcg@10	0.118
cosine_ndcg@100	0.1733
cosine_mrr@10	0.0913
cosine_map@100	0.1008

Training Details

Training Dataset

Unnamed Dataset

• Size: 9,000 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 116 tokens mean: 207.16 tokens max: 288 tokens	min: 4 tokens mean: 15.73 tokens max: 37 tokens

• Samples:

  positive	anchor
  ation. US 10-year Treasury yields have risen more than 30 basis points since the Sep 20 FOMC meeting, but just 5 basis points for 2-year notes, steepening the yield curve. Most importantly, the recent rise in rates was a “break out” for yields above post-COVID expansion highs (Figure 4). Figure 4: 10-year US Nominal Treasury yield and 10-year Inflation Indexed US Treasury Source: Haver Analytics as of September 28, 2023. Gray areas are recessions. Past performance is no guarantee of future results. Real results may vary. Citi Global Wealth Investments	CIO Strategy Bulletin
  extension How to Save Money Using the Citi Shop Extension	Citi.com Save money online shopping Save shopping online How to save money shopping online How to Save Money Shopping Online
  common control of Citigroup. Outside the U.S., investment products and services are provided by other Citigroup affiliates. Investment Management services (including portfolio management) are available through CGMI, CGA, Citibank, N.A. and other affiliated advisory businesses. These Citigroup affiliates, including CGA, will be compensated for the respective investment management, advisory, administrative, distribution and placement services they may provide. International Personal Bank U.S. (“IPB U.S.”) is a business of Citigroup which provides its clients access to a broad array of products and services available through Citigroup, its bank and non-bank affiliates worldwide (collectively, “Citi”). Through IPB U.S. prospects and clients have access to the Citigold® Private Client International, Citigold® International, International Personal, Citi Global Executive Preferred, and Citi Global Executive Account Packages. Investment products and services are made available through Citi Personal Investments International (“CPII”), a business	What are the typical assumpitons given in the report?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768
      ],
      "matryoshka_weights": [
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_map@100
0.0355	10	2.0527	-
0.0709	20	2.3092	-
0.1064	30	1.8688	-
0.1418	40	1.8818	-
0.1773	50	1.75	-
0.2128	60	1.8462	-
0.2482	70	1.7534	-
0.2837	80	1.7534	-
0.3191	90	1.7454	-
0.3546	100	1.7037	-
0.3901	110	1.6765	-
0.4255	120	1.5392	-
0.4610	130	1.722	-
0.4965	140	1.5609	-
0.5319	150	1.6001	-
0.5674	160	1.5694	-
0.6028	170	1.7528	-
0.6383	180	1.5393	-
0.6738	190	1.6765	-
0.7092	200	1.4197	-
0.7447	210	1.5231	-
0.7801	220	1.7733	-
0.8156	230	1.5464	-
0.8511	240	1.5321	-
0.8865	250	1.5727	-
0.9220	260	1.5909	-
0.9574	270	1.6485	-
0.9929	280	1.6605	-
1.0	282	-	0.1008

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.14
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.1.2+cu121
• Accelerate: 0.32.1
• Datasets: 2.19.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}