metadata

base_model: Snowflake/snowflake-arctic-embed-m
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
  - dot_accuracy@1
  - dot_accuracy@3
  - dot_accuracy@5
  - dot_accuracy@10
  - dot_precision@1
  - dot_precision@3
  - dot_precision@5
  - dot_precision@10
  - dot_recall@1
  - dot_recall@3
  - dot_recall@5
  - dot_recall@10
  - dot_ndcg@10
  - dot_mrr@10
  - dot_map@100
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:568
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: >-
      What measures did the device manufacturer take to protect individuals from
      unwanted tracking?
    sentences:
      - >-
        Tailored to the target of the explanation. Explanations should be
        targeted to specific audiences and clearly state that audience. An
        explanation provided to the subject of a decision might differ from one
        provided to an advocate, or to a domain expert or decision maker.
        Tailoring should be assessed (e.g., via user experience research). 

        43
              NOTICE & 
        EXPLANATION 

        WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS

        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional 

        technical standards and practices that are tailored for particular
        sectors and contexts. 

        Tailored to the level of risk. An assessment should be done to determine
        the level of risk of the auto -
      - >-
        7

        • A device originally developed to help people track and find lost items
        has been used as a tool by stalkers to trackvictims’ locations in
        violation of their privacy and safet y. The device manufacturer took
        steps after release to

        protect people from unwanted tracking by alerting people on their phones
        when a device is found to be movingwith them over time and also by
        having the device make an occasional noise, but not all phones are
        ableto receive the notification and the devices remain a safety concern
        due to their misuse.

        8
      - >-
        -

        sonable expectations in a given context and with a focus on ensuring
        broad accessibility and protecting the public from especially harm

        -

        ful impacts. In some cases, a human or other alternative may be re -

        quired by law. You should have access to timely human consider -

        ation and remedy by a fallback and escalation process if an automat -

        ed system fails, it produces an error, or you would like to appeal or
        contest its impacts on you. Human consideration and fallback should be
        accessible, equitable, effective, maintained, accompanied by appropriate
        operator training, and should not impose an unrea

        -
  - source_sentence: >-
      Why is ongoing monitoring and mitigation important for automated systems
      after deployment?
    sentences:
      - >-
        -

        test its impacts on you 

        Proportionate. The availability of human consideration and fallback,
        along with associated training and 

        safeguards against human bias, should be proportionate to the potential
        of the automated system to meaning -

        fully impact rights, opportunities, or access. Automated systems that
        have greater control over outcomes, provide input to high-stakes
        decisions, relate to sensitive domains, or otherwise have a greater
        potential to meaningfully impact rights, opportunities, or access should
        have greater availability (e.g., staffing) and over

        -

        sight of human consideration and fallback mechanisms. 

        Accessible. Mechanisms for human consideration and fallback, whether
        in-person, on paper, by phone, or
      - >-
        algorithmic discrimination, avoid meaningful harm, and achieve equity
        goals. 

        Ongoing monitoring and mitigation. Automated systems should be regularly
        monitored to assess algo -

        rithmic discrimination that might arise from unforeseen interactions of
        the system with inequities not accounted for during the pre-deployment
        testing, changes to the system after deployment, or changes to the
        context of use or associated data. Monitoring and disparity assessment
        should be performed by the entity deploying or using the automated
        system to examine whether the system has led to algorithmic discrimina

        -
      - >-
        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional 

        technical standards and practices that are tailored for particular
        sectors and contexts. 

        Ongoing monitoring. Automated systems should have ongoing monitoring
        procedures, including recalibra -

        tion procedures, in place to ensure that their performance does not fall
        below an acceptable level over time, 

        based on changing real-world conditions or deployment contexts,
        post-deployment modification, or unexpect -
  - source_sentence: >-
      What should be included in the measurement of the impact of risks
      associated with automated systems?
    sentences:
      - >-
        104 

        48
              HUMAN ALTERNATIVES, 
        CONSIDERATION, AND 

        FALLBACK 

        WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS

        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional 

        technical standards and practices that are tailored for particular
        sectors and contexts. 

        An automated system should provide demonstrably effective mechanisms to
        opt out in favor of a human alterna -

        tive, where appropriate, as well as timely human consideration and
        remedy by a fallback system, with additional 

        human oversight and safeguards for systems used in sensitive domains,
        and with training and assessment for any human-based portions of the
        system to ensure effectiveness.
      - >-
        collection and use is legal and consistent with the expectations of the
        people whose data is collected. User experience research should be
        conducted to confirm that people understand what data is being collected
        about them and how it will be used, and that this collection matches
        their expectations and desires.
      - >-
        -

        surement of the impact of risks should be included and balanced such
        that high impact risks receive attention and mitigation proportionate
        with those impacts. Automated systems with the intended purpose of
        violating the safety of others should not be developed or used; systems
        with such safety violations as identified unin

        -

        tended consequences should not be used until the risk can be mitigated.
        Ongoing risk mitigation may necessi -

        tate rollback or significant modification to a launched automated
        system. 

        18
              
         
         
         
         
          SAFE AND EFFECTIVE 
        SYSTEMS 

        WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS

        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional
  - source_sentence: >-
      What measures should be taken to avoid "mission creep" when identifying
      goals for data collection?
    sentences:
      - >-
        narrow identified goals, to avoid "mission creep."  Anticipated data
        collection should be determined to be strictly necessary to the
        identified goals and should be minimized as much as possible. Data
        collected based on these identified goals and for a specific context
        should not be used in a different context without assessing for new
        privacy risks and implementing appropriate mitigation measures, which
        may include express consent. Clear timelines for data retention should
        be established, with data deleted as soon as possible in accordance with
        legal or policy-based limitations. Determined data retention timelines
        should be documented and justi

        -

        fied.
      - >-
        with more and more companies tracking the behavior of the American
        public, building individual profiles based on this data, and using this
        granular-level information as input into automated systems that further
        track, profile, and impact the American public. Government agencies,
        particularly law enforcement agencies, also use and help develop a
        variety of technologies that enhance and expand surveillance
        capabilities, which similarly collect data used as input into other
        automated systems that directly impact people’s lives. Federal law has
        not grown to address the expanding scale of private data collection, or
        of the ability of governments at all levels to access that data and
        leverage the means of private collection.
      - >-
        additional technical standards and practices that should be tailored for
        particular sectors and contexts. While 

        existing laws informed the development of the Blueprint for an AI Bill
        of Rights, this framework does not detail those laws beyond providing
        them as examples, where appropriate, of existing protective measures.
        This framework instead shares a broad, forward-leaning vision of
        recommended principles for automated system development and use to
        inform private and public involvement with these systems where they have
        the poten-tial to meaningfully impact rights, opportunities, or access.
        Additionall y, this framework does not analyze or
  - source_sentence: >-
      What types of data are considered sensitive according to the context
      provided?
    sentences:
      - >-
        Provide the public with mechanisms for appropriate and meaningful
        consent, access, and 

        control over their data 

        Use-specific consent. Consent practices should not allow for abusive
        surveillance practices. Where data 

        collectors or automated systems seek consent, they should seek it for
        specific, narrow use contexts, for specif -

        ic time durations, and for use by specific entities. Consent should not
        extend if any of these conditions change; consent should be re-acquired
        before using data if the use case changes, a time limit elapses, or data
        is trans

        -
      - >-
        and home, work, or school environmental data); or have the reasonable
        potential to be used in ways that are likely to expose individuals to
        meaningful harm, such as a loss of privacy or financial harm due to
        identity theft. Data and metadata generated by or about those who are
        not yet legal adults is also sensitive, even if not related to a
        sensitive domain. Such data includes, but is not limited to, numerical,
        text, image, audio, or video data. “Sensitive domains” are those in
        which activities being conducted can cause material harms, including
        signifi
      - >-
        that data to inform the results of the automated system and why such use
        will not violate any applicable laws. 

        In cases of high-dimensional and/or derived attributes, such
        justifications can be provided as overall 

        descriptions of the attribute generation process and appropriateness. 

        19
               
         
          SAFE AND EFFECTIVE 
        SYSTEMS 

        WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS

        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional 

        technical standards and practices that are tailored for particular
        sectors and contexts. 

        Derived data sources tracked and reviewed carefully. Data that is
        derived from other data through
model-index:
  - name: SentenceTransformer based on Snowflake/snowflake-arctic-embed-m
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: Unknown
          type: unknown
        metrics:
          - type: cosine_accuracy@1
            value: 0.7677725118483413
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8862559241706162
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9241706161137441
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.981042654028436
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7677725118483413
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.29541864139020535
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.1848341232227488
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.0981042654028436
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7677725118483413
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8862559241706162
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9241706161137441
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.981042654028436
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8716745978729181
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.8371304445948993
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.838229587684564
            name: Cosine Map@100
          - type: dot_accuracy@1
            value: 0.7677725118483413
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.8862559241706162
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.9241706161137441
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.981042654028436
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.7677725118483413
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.29541864139020535
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.1848341232227488
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.0981042654028436
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.7677725118483413
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.8862559241706162
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.9241706161137441
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.981042654028436
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.8716745978729181
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.8371304445948993
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.838229587684564
            name: Dot Map@100

SentenceTransformer based on Snowflake/snowflake-arctic-embed-m

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: Snowflake/snowflake-arctic-embed-m
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 tokens
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What types of data are considered sensitive according to the context provided?',
    'and home, work, or school environmental data); or have the reasonable potential to be used in ways that are likely to expose individuals to meaningful harm, such as a loss of privacy or financial harm due to identity theft. Data and metadata generated by or about those who are not yet legal adults is also sensitive, even if not related to a sensitive domain. Such data includes, but is not limited to, numerical, text, image, audio, or video data. “Sensitive domains” are those in which activities being conducted can cause material harms, including signifi',
    'Provide the public with mechanisms for appropriate and meaningful consent, access, and \ncontrol over their data \nUse-specific consent. Consent practices should not allow for abusive surveillance practices. Where data \ncollectors or automated systems seek consent, they should seek it for specific, narrow use contexts, for specif -\nic time durations, and for use by specific entities. Consent should not extend if any of these conditions change; consent should be re-acquired before using data if the use case changes, a time limit elapses, or data is trans\n-',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7678
cosine_accuracy@3	0.8863
cosine_accuracy@5	0.9242
cosine_accuracy@10	0.981
cosine_precision@1	0.7678
cosine_precision@3	0.2954
cosine_precision@5	0.1848
cosine_precision@10	0.0981
cosine_recall@1	0.7678
cosine_recall@3	0.8863
cosine_recall@5	0.9242
cosine_recall@10	0.981
cosine_ndcg@10	0.8717
cosine_mrr@10	0.8371
cosine_map@100	0.8382
dot_accuracy@1	0.7678
dot_accuracy@3	0.8863
dot_accuracy@5	0.9242
dot_accuracy@10	0.981
dot_precision@1	0.7678
dot_precision@3	0.2954
dot_precision@5	0.1848
dot_precision@10	0.0981
dot_recall@1	0.7678
dot_recall@3	0.8863
dot_recall@5	0.9242
dot_recall@10	0.981
dot_ndcg@10	0.8717
dot_mrr@10	0.8371
dot_map@100	0.8382

Training Details

Training Dataset

Unnamed Dataset

Size: 568 training samples
Columns: sentence_0 and sentence_1
Approximate statistics based on the first 568 samples:
sentence_0 sentence_1
type string string
details
min: 11 tokens
mean: 19.09 tokens
max: 36 tokens

min: 22 tokens
mean: 118.73 tokens
max: 160 tokens

	sentence_0	sentence_1
type	string	string
details	min: 11 tokens mean: 19.09 tokens max: 36 tokens	min: 22 tokens mean: 118.73 tokens max: 160 tokens

Samples:

sentence_0	sentence_1
`What is the purpose of the AI Bill of Rights mentioned in the context?`	`BLUEPRINT FOR AN AI B ILL OF RIGHTS MAKING AUTOMATED SYSTEMS WORK FOR THE AMERICAN PEOPLE OCTOBER 2022`
`When was the Blueprint for an AI Bill of Rights published?`	`BLUEPRINT FOR AN AI B ILL OF RIGHTS MAKING AUTOMATED SYSTEMS WORK FOR THE AMERICAN PEOPLE OCTOBER 2022`
`What is the purpose of the Blueprint for an AI Bill of Rights published by the White House Office of Science and Technology Policy?`	About this Document The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People was published by the White House Office of Science and Technology Policy in October 2022. This framework was released one year after OSTP announced the launch of a process to develop “a bill of rights for an AI-powered world.” Its release follows a year of public engagement to inform this initiative. The framework is available online at: https://www.whitehouse.gov/ostp/ai-bill-of-rights About the Office of Science and Technology Policy The Office of Science and Technology Policy (OSTP) was established by the National Science and Technology

Loss: MatryoshkaLoss with these parameters:

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 20
per_device_eval_batch_size: 20
num_train_epochs: 10
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 20
per_device_eval_batch_size: 20
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 10
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
eval_use_gather_object: False
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	cosine_map@100
1.0	29	0.7800
1.7241	50	0.8242
2.0	58	0.8382

Framework Versions

Python: 3.10.12
Sentence Transformers: 3.1.1
Transformers: 4.44.2
PyTorch: 2.4.1+cu121
Accelerate: 0.34.2
Datasets: 2.19.2
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}