--- language: - en license: apache-2.0 tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:6300 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss base_model: BAAI/bge-base-en-v1.5 widget: - source_sentence: Goodwill is recognized for the excess of the purchase price over the fair value of tangible and identifiable intangible net assets of businesses acquired. In evaluating goodwill impairment, a qualitative assessment is performed to determine the likelihood that the fair value of a reporting unit is less than its carrying amount. This might lead to further testing of goodwill for impairment, which includes comparing the fair value of the reporting unit to its carrying value (including attributable goodwill). Fair value for our reporting units is determined using an income or market approach incorporating market participant considerations and management’s assumptions. sentences: - How is goodwill reviewed for impairment in a company, and what methods are used to determine the fair value of reporting units? - What regulatory framework does the FCC currently apply to broadband internet access services as of 2023? - What were the total interest payments made by the company in 2023, 2022, and 2021? - source_sentence: Part IV Item 15, titled 'Exhibits, Financial Statement Schedules', includes the 'Index to Financial Statements' and the 'Index to Financial Statement Schedules.' sentences: - What was the net cash from operations reported for the year ended June 30, 2023? - Where would one find the 'Index to Financial Statements' and the 'Index to Financial Statement Schedules' mentioned? - What is the trajectory of the AMPTC for microinverters starting in 2030? - source_sentence: As of December 31, 2023, the total amortized cost, net of valuation allowance, for non-U.S. government securities amounted to $14,516 million. sentences: - How much federal net operating loss carryforwards did the company have at the end of 2023, and how much of it is expected to be realized? - What accounting principles are followed in the preparation of Goldman Sachs' consolidated financial statements for 2023? - What was the total amortized cost, net of valuation allowance, for non-U.S. government securities as of December 31, 2023? - source_sentence: Information about legal proceedings in the Annual Report on Form 10-K is incorporated by reference under several notes and sections. sentences: - What method is used to provide information about legal proceedings in the Annual Report on Form 10-K? - What can cause delays in pharmaceutical product launches? - What was the total amount of cash dividends declared by the company per share in the fiscal year ending on October 1, 2023? - source_sentence: MERS database revenues contain multiple performance obligations related to each new loan registration and future transfers, and the revenues are primarily recorded at the point in time of each transaction. sentences: - How are revenues from MERS database recognized? - How did The Home Depot, Inc.'s basic earnings per share change from 2020 to 2022? - How many active sellers and buyers did Etsy's marketplaces connect as of December 31, 2023? pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 model-index: - name: BGE base Financial Matryoshka results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.7228571428571429 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8528571428571429 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.89 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9185714285714286 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.7228571428571429 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2842857142857143 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17799999999999996 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09185714285714283 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.7228571428571429 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8528571428571429 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.89 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9185714285714286 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.8244010006831627 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7936836734693877 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7971656786986449 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 512 type: dim_512 metrics: - type: cosine_accuracy@1 value: 0.7157142857142857 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8471428571428572 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8857142857142857 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9185714285714286 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.7157142857142857 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.28238095238095234 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17714285714285713 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09185714285714283 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.7157142857142857 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8471428571428572 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8857142857142857 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9185714285714286 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.8209116379330612 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7891343537414967 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7926472335071902 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.71 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8428571428571429 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8757142857142857 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9128571428571428 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.71 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.28095238095238095 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17514285714285713 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09128571428571428 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.71 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8428571428571429 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8757142857142857 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9128571428571428 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.8138965576076403 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7818429705215417 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7855894139852542 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.6871428571428572 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8185714285714286 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8628571428571429 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8971428571428571 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6871428571428572 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.27285714285714285 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17257142857142854 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.0897142857142857 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6871428571428572 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8185714285714286 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8628571428571429 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.8971428571428571 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7953389524625682 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7622392290249432 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7667451557566504 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 64 type: dim_64 metrics: - type: cosine_accuracy@1 value: 0.6628571428571428 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.7842857142857143 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8371428571428572 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8771428571428571 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6628571428571428 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.26142857142857145 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.1674285714285714 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.0877142857142857 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6628571428571428 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.7842857142857143 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8371428571428572 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.8771428571428571 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7701231991584621 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7357777777777779 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7410692697767751 name: Cosine Map@100 --- # BGE base Financial Matryoshka This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - json - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("chcho/bge-base-financial-matryoshka") # Run inference sentences = [ 'MERS database revenues contain multiple performance obligations related to each new loan registration and future transfers, and the revenues are primarily recorded at the point in time of each transaction.', 'How are revenues from MERS database recognized?', "How many active sellers and buyers did Etsy's marketplaces connect as of December 31, 2023?", ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Datasets: `dim_768`, `dim_512`, `dim_256`, `dim_128` and `dim_64` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | dim_768 | dim_512 | dim_256 | dim_128 | dim_64 | |:--------------------|:-----------|:-----------|:-----------|:-----------|:-----------| | cosine_accuracy@1 | 0.7229 | 0.7157 | 0.71 | 0.6871 | 0.6629 | | cosine_accuracy@3 | 0.8529 | 0.8471 | 0.8429 | 0.8186 | 0.7843 | | cosine_accuracy@5 | 0.89 | 0.8857 | 0.8757 | 0.8629 | 0.8371 | | cosine_accuracy@10 | 0.9186 | 0.9186 | 0.9129 | 0.8971 | 0.8771 | | cosine_precision@1 | 0.7229 | 0.7157 | 0.71 | 0.6871 | 0.6629 | | cosine_precision@3 | 0.2843 | 0.2824 | 0.281 | 0.2729 | 0.2614 | | cosine_precision@5 | 0.178 | 0.1771 | 0.1751 | 0.1726 | 0.1674 | | cosine_precision@10 | 0.0919 | 0.0919 | 0.0913 | 0.0897 | 0.0877 | | cosine_recall@1 | 0.7229 | 0.7157 | 0.71 | 0.6871 | 0.6629 | | cosine_recall@3 | 0.8529 | 0.8471 | 0.8429 | 0.8186 | 0.7843 | | cosine_recall@5 | 0.89 | 0.8857 | 0.8757 | 0.8629 | 0.8371 | | cosine_recall@10 | 0.9186 | 0.9186 | 0.9129 | 0.8971 | 0.8771 | | **cosine_ndcg@10** | **0.8244** | **0.8209** | **0.8139** | **0.7953** | **0.7701** | | cosine_mrr@10 | 0.7937 | 0.7891 | 0.7818 | 0.7622 | 0.7358 | | cosine_map@100 | 0.7972 | 0.7926 | 0.7856 | 0.7667 | 0.7411 | ## Training Details ### Training Dataset #### json * Dataset: json * Size: 6,300 training samples * Columns: positive and anchor * Approximate statistics based on the first 1000 samples: | | positive | anchor | |:--------|:-----------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | positive | anchor | |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------| | We use a variety of methodologies to determine the fair value of these assets, including discounted cash flow models, which include assumptions we believe are consistent with those a market participant would use. | How is the fair value of intangible assets determined within a company? | | We continue to own a 35% minority ownership in Gentiva Hospice operations after it was restructured into a new stand-alone company. | What percentage minority ownership does the company retain in Gentiva Hospice after the restructuring? | | The net interest income for the first quarter of 2023 was $14,448 million. | What was the net interest income for the first quarter of 2023? | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `gradient_accumulation_steps`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 4 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `bf16`: True - `tf32`: True - `load_best_model_at_end`: True - `optim`: adamw_torch_fused - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 16 - `eval_accumulation_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 4 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: True - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 | |:----------:|:------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:| | 0.8122 | 10 | 1.5791 | - | - | - | - | - | | 0.9746 | 12 | - | 0.8089 | 0.8028 | 0.7958 | 0.7714 | 0.7428 | | 1.6244 | 20 | 0.6637 | - | - | - | - | - | | 1.9492 | 24 | - | 0.8209 | 0.8166 | 0.8109 | 0.7913 | 0.7615 | | 2.4365 | 30 | 0.5072 | - | - | - | - | - | | **2.9239** | **36** | **-** | **0.8229** | **0.82** | **0.8133** | **0.7959** | **0.7704** | | 3.2487 | 40 | 0.394 | - | - | - | - | - | | 3.8985 | 48 | - | 0.8244 | 0.8209 | 0.8139 | 0.7953 | 0.7701 | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.9.5 - Sentence Transformers: 3.3.1 - Transformers: 4.41.2 - PyTorch: 2.1.2+cu121 - Accelerate: 0.27.2 - Datasets: 2.19.1 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```