Hugging Face's logo

jet-taekyo
/
snowflake_finetuned_recursive

Sentence Similarity
sentence-transformers
Safetensors
bert
feature-extraction
Generated from Trainer
dataset_size:714
loss:MatryoshkaLoss
loss:MultipleNegativesRankingLoss
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
snowflake_finetuned_recursive / README.md
jet-taekyo's picture
jet-taekyo
Add new SentenceTransformer model.
ef3de2b verified
preview code
|
raw
history blame contribute delete
37.7 kB
---
base_model: Snowflake/snowflake-arctic-embed-m
library_name: sentence-transformers
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
- dot_accuracy@1
- dot_accuracy@3
- dot_accuracy@5
- dot_accuracy@10
- dot_precision@1
- dot_precision@3
- dot_precision@5
- dot_precision@10
- dot_recall@1
- dot_recall@3
- dot_recall@5
- dot_recall@10
- dot_ndcg@10
- dot_mrr@10
- dot_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:714
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: What are some examples of data privacy issues mentioned in the
context?
sentences:
- "on a principle of local control, such that those individuals closest to the data\
\ subject have more access while \nthose who are less proximate do not (e.g.,\
\ a teacher has access to their students’ daily progress data while a \nsuperintendent\
\ does not). \nReporting. In addition to the reporting on data privacy (as listed\
\ above for non-sensitive data), entities devel-\noping technologies related to\
\ a sensitive domain and those collecting, using, storing, or sharing sensitive\
\ data \nshould, whenever appropriate, regularly provide public reports describing:\
\ any data security lapses or breaches \nthat resulted in sensitive data leaks;\
\ the number, type, and outcomes of ethical pre-reviews undertaken; a \ndescription\
\ of any data sold, shared, or made public, and how that data was assessed to\
\ determine it did not pres-\nent a sensitive data risk; and ongoing risk identification\
\ and management procedures, and any mitigation added"
- "DATA PRIVACY \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples\
\ of how these principles can become reality, through laws, policies, and practical\
\ \ntechnical and sociotechnical approaches to protecting rights, opportunities,\
\ and access. \nThe Privacy Act of 1974 requires privacy protections for personal\
\ information in federal \nrecords systems, including limits on data retention,\
\ and also provides individuals a general \nright to access and correct their\
\ data. Among other things, the Privacy Act limits the storage of individual \n\
information in federal systems of records, illustrating the principle of limiting\
\ the scope of data retention. Under \nthe Privacy Act, federal agencies may only\
\ retain data about an individual that is “relevant and necessary” to \naccomplish\
\ an agency’s statutory purpose or to comply with an Executive Order of the President.\
\ The law allows"
- "DATA PRIVACY \nWHY THIS PRINCIPLE IS IMPORTANT\nThis section provides a brief\
\ summary of the problems which the principle seeks to address and protect \n\
against, including illustrative examples. \n•\nAn insurer might collect data from\
\ a person's social media presence as part of deciding what life\ninsurance rates\
\ they should be offered.64\n•\nA data broker harvested large amounts of personal\
\ data and then suffered a breach, exposing hundreds of\nthousands of people to\
\ potential identity theft. 65\n•\nA local public housing authority installed\
\ a facial recognition system at the entrance to housing complexes to\nassist\
\ law enforcement with identifying individuals viewed via camera when police reports\
\ are filed, leading\nthe community, both those living in the housing complex\
\ and not, to have videos of them sent to the local\npolice department and made\
\ available for scanning by its facial recognition software.66\n•"
- source_sentence: What are the main topics covered in the National Institute of Standards
and Technology's AI Risk Management Framework?
sentences:
- "https://www.rand.org/pubs/research_reports/RRA2977-2.html. \nNicoletti, L. et\
\ al. (2023) Humans Are Biased. Generative Ai Is Even Worse. Bloomberg. \nhttps://www.bloomberg.com/graphics/2023-generative-ai-bias/.\
\ \nNational Institute of Standards and Technology (2024) Adversarial Machine\
\ Learning: A Taxonomy and \nTerminology of Attacks and Mitigations https://csrc.nist.gov/pubs/ai/100/2/e2023/final\
\ \nNational Institute of Standards and Technology (2023) AI Risk Management Framework.\
\ \nhttps://www.nist.gov/itl/ai-risk-management-framework \nNational Institute\
\ of Standards and Technology (2023) AI Risk Management Framework, Chapter 3:\
\ AI \nRisks and Trustworthiness. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/3-sec-characteristics\
\ \nNational Institute of Standards and Technology (2023) AI Risk Management Framework,\
\ Chapter 6: AI \nRMF Profiles. https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Core_And_Profiles/6-sec-profile"
- "(e.g., via red-teaming, field testing, participatory engagements, performance\
\ \nassessments, user feedback mechanisms). \nHuman-AI Configuration \nAI Actor\
\ Tasks: AI Development, AI Deployment, AI Impact Assessment, Operation and Monitoring\
\ \n \nMANAGE 2.2: Mechanisms are in place and applied to sustain the value of\
\ deployed AI systems. \nAction ID \nSuggested Action \nGAI Risks \nMG-2.2-001\
\ \nCompare GAI system outputs against pre-defined organization risk tolerance,\
\ \nguidelines, and principles, and review and test AI-generated content against\
\ \nthese guidelines. \nCBRN Information or Capabilities; \nObscene, Degrading,\
\ and/or \nAbusive Content; Harmful Bias and \nHomogenization; Dangerous, \nViolent,\
\ or Hateful Content \nMG-2.2-002 \nDocument training data sources to trace the\
\ origin and provenance of AI-\ngenerated content. \nInformation Integrity \n\
MG-2.2-003 \nEvaluate feedback loops between GAI system content provenance and\
\ human"
- "domain or for functions that are required for administrative reasons (e.g., school\
\ attendance records), unless \nconsent is acquired, if appropriate, and the additional\
\ expectations in this section are met. Consent for non-\nnecessary functions\
\ should be optional, i.e., should not be required, incentivized, or coerced in\
\ order to \nreceive opportunities or access to services. In cases where data\
\ is provided to an entity (e.g., health insurance \ncompany) in order to facilitate\
\ payment for such a need, that data should only be used for that purpose. \n\
Ethical review and use prohibitions. Any use of sensitive data or decision process\
\ based in part on sensi-\ntive data that might limit rights, opportunities, or\
\ access, whether the decision is automated or not, should go \nthrough a thorough\
\ ethical review and monitoring, both in advance and by periodic review (e.g.,\
\ via an indepen-\ndent ethics committee or similarly robust process). In some\
\ cases, this ethical review may determine that data"
- source_sentence: How can organizations leverage user feedback to enhance content
provenance and risk management efforts?
sentences:
- "tested, there will always be situations for which the system fails. The American\
\ public deserves protection via human \nreview against these outlying or unexpected\
\ scenarios. In the case of time-critical systems, the public should not have\
\ \nto wait—immediate human consideration and fallback should be available. In\
\ many time-critical systems, such a \nremedy is already immediately available,\
\ such as a building manager who can open a door in the case an automated \ncard\
\ access system fails. \nIn the criminal justice system, employment, education,\
\ healthcare, and other sensitive domains, automated systems \nare used for many\
\ purposes, from pre-trial risk assessments and parole decisions to technologies\
\ that help doctors \ndiagnose disease. Absent appropriate safeguards, these technologies\
\ can lead to unfair, inaccurate, or dangerous \noutcomes. These sensitive domains\
\ require extra protections. It is critically important that there is extensive\
\ human \noversight in such settings."
- "enable organizations to maximize the utility of provenance data and risk management\
\ efforts. \nA.1.7. Enhancing Content Provenance through Structured Public Feedback\
\ \nWhile indirect feedback methods such as automated error collection systems\
\ are useful, they often lack \nthe context and depth that direct input from end\
\ users can provide. Organizations can leverage feedback \napproaches described\
\ in the Pre-Deployment Testing section to capture input from external sources\
\ such \nas through AI red-teaming. \nIntegrating pre- and post-deployment external\
\ feedback into the monitoring process for GAI models and \ncorresponding applications\
\ can help enhance awareness of performance changes and mitigate potential \n\
risks and harms from outputs. There are many ways to capture and make use of user\
\ feedback – before \nand after GAI systems and digital content transparency approaches\
\ are deployed – to gain insights about"
- "A.1. Governance \nA.1.1. Overview \nLike any other technology system, governance\
\ principles and techniques can be used to manage risks \nrelated to generative\
\ AI models, capabilities, and applications. Organizations may choose to apply\
\ their \nexisting risk tiering to GAI systems, or they may opt to revise or update\
\ AI system risk levels to address \nthese unique GAI risks. This section describes\
\ how organizational governance regimes may be re-\nevaluated and adjusted for\
\ GAI contexts. It also addresses third-party considerations for governing across\
\ \nthe AI value chain. \nA.1.2. Organizational Governance \nGAI opportunities,\
\ risks and long-term performance characteristics are typically less well-understood\
\ \nthan non-generative AI tools and may be perceived and acted upon by humans\
\ in ways that vary greatly. \nAccordingly, GAI may call for different levels of\
\ oversight from AI Actors or different human-AI"
- source_sentence: What should be ensured for users who have trouble with the automated
system?
sentences:
- "32 \nMEASURE 2.6: The AI system is evaluated regularly for safety risks – as\
\ identified in the MAP function. The AI system to be \ndeployed is demonstrated\
\ to be safe, its residual negative risk does not exceed the risk tolerance, and\
\ it can fail safely, particularly if \nmade to operate beyond its knowledge limits.\
\ Safety metrics reflect system reliability and robustness, real-time monitoring,\
\ and \nresponse times for AI system failures. \nAction ID \nSuggested Action\
\ \nGAI Risks \nMS-2.6-001 \nAssess adverse impacts, including health and wellbeing\
\ impacts for value chain \nor other AI Actors that are exposed to sexually explicit,\
\ offensive, or violent \ninformation during GAI training and maintenance. \nHuman-AI\
\ Configuration; Obscene, \nDegrading, and/or Abusive \nContent; Value Chain and\
\ \nComponent Integration; \nDangerous, Violent, or Hateful \nContent \nMS-2.6-002\
\ \nAssess existence or levels of harmful bias, intellectual property infringement,"
- "APPENDIX\nSystems that impact the safety of communities such as automated traffic\
\ control systems, elec \n-ctrical grid controls, smart city technologies, and\
\ industrial emissions and environmental\nimpact control algorithms; and\nSystems\
\ related to access to benefits or services or assignment of penalties such as\
\ systems that\nsupport decision-makers who adjudicate benefits such as collating\
\ or analyzing information or\nmatching records, systems which similarly assist\
\ in the adjudication of administrative or criminal\npenalties, fraud detection\
\ algorithms, services or benefits access control algorithms, biometric\nsystems\
\ used as access control, and systems which make benefits or services related\
\ decisions on a\nfully or partially autonomous basis (such as a determination\
\ to revoke benefits).\n54"
- "meaningfully impact rights, opportunities, or access should have greater availability\
\ (e.g., staffing) and over­\nsight of human consideration and fallback mechanisms.\
\ \nAccessible. Mechanisms for human consideration and fallback, whether in-person,\
\ on paper, by phone, or \notherwise provided, should be easy to find and use.\
\ These mechanisms should be tested to ensure that users \nwho have trouble with\
\ the automated system are able to use human consideration and fallback, with\
\ the under­\nstanding that it may be these users who are most likely to need\
\ the human assistance. Similarly, it should be \ntested to ensure that users\
\ with disabilities are able to find and use human consideration and fallback\
\ and also \nrequest reasonable accommodations or modifications. \nConvenient.\
\ Mechanisms for human consideration and fallback should not be unreasonably burdensome\
\ as \ncompared to the automated system’s equivalent. \n49"
- source_sentence: What must lenders provide to consumers who are denied credit under
the Fair Credit Reporting Act?
sentences:
- "8 \nTrustworthy AI Characteristics: Accountable and Transparent, Privacy Enhanced,\
\ Safe, Secure and \nResilient \n2.5. Environmental Impacts \nTraining, maintaining,\
\ and operating (running inference on) GAI systems are resource-intensive activities,\
\ \nwith potentially large energy and environmental footprints. Energy and carbon\
\ emissions vary based on \nwhat is being done with the GAI model (i.e., pre-training,\
\ fine-tuning, inference), the modality of the \ncontent, hardware used, and type\
\ of task or application. \nCurrent estimates suggest that training a single transformer\
\ LLM can emit as much carbon as 300 round-\ntrip flights between San Francisco\
\ and New York. In a study comparing energy consumption and carbon \nemissions\
\ for LLM inference, generative tasks (e.g., text summarization) were found to\
\ be more energy- \nand carbon-intensive than discriminative or non-generative\
\ tasks (e.g., text classification)."
- "that consumers who are denied credit receive \"adverse action\" notices. Anyone\
\ who relies on the information in a \ncredit report to deny a consumer credit\
\ must, under the Fair Credit Reporting Act, provide an \"adverse action\" \n\
notice to the consumer, which includes \"notice of the reasons a creditor took\
\ adverse action on the application \nor on an existing credit account.\"90 In\
\ addition, under the risk-based pricing rule,91 lenders must either inform \n\
borrowers of their credit score, or else tell consumers when \"they are getting\
\ worse terms because of \ninformation in their credit report.\" The CFPB has\
\ also asserted that \"[t]he law gives every applicant the right to \na specific\
\ explanation if their application for credit was denied, and that right is not\
\ diminished simply because \na company uses a complex algorithm that it doesn't\
\ understand.\"92 Such explanations illustrate a shared value \nthat certain decisions\
\ need to be explained."
- "measures to prevent, flag, or take other action in response to outputs that \n\
reproduce particular training data (e.g., plagiarized, trademarked, patented,\
\ \nlicensed content or trade secret material). \nIntellectual Property; CBRN\
\ \nInformation or Capabilities"
model-index:
- name: SentenceTransformer based on Snowflake/snowflake-arctic-embed-m
results:
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: Unknown
type: unknown
metrics:
- type: cosine_accuracy@1
value: 0.881578947368421
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.9671052631578947
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.9868421052631579
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 1.0
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.881578947368421
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.3223684210526316
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.19736842105263155
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.09999999999999999
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.881578947368421
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.9671052631578947
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.9868421052631579
name: Cosine Recall@5
- type: cosine_recall@10
value: 1.0
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.9460063349721777
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.9282346491228071
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.9282346491228068
name: Cosine Map@100
- type: dot_accuracy@1
value: 0.881578947368421
name: Dot Accuracy@1
- type: dot_accuracy@3
value: 0.9671052631578947
name: Dot Accuracy@3
- type: dot_accuracy@5
value: 0.9868421052631579
name: Dot Accuracy@5
- type: dot_accuracy@10
value: 1.0
name: Dot Accuracy@10
- type: dot_precision@1
value: 0.881578947368421
name: Dot Precision@1
- type: dot_precision@3
value: 0.3223684210526316
name: Dot Precision@3
- type: dot_precision@5
value: 0.19736842105263155
name: Dot Precision@5
- type: dot_precision@10
value: 0.09999999999999999
name: Dot Precision@10
- type: dot_recall@1
value: 0.881578947368421
name: Dot Recall@1
- type: dot_recall@3
value: 0.9671052631578947
name: Dot Recall@3
- type: dot_recall@5
value: 0.9868421052631579
name: Dot Recall@5
- type: dot_recall@10
value: 1.0
name: Dot Recall@10
- type: dot_ndcg@10
value: 0.9460063349721777
name: Dot Ndcg@10
- type: dot_mrr@10
value: 0.9282346491228071
name: Dot Mrr@10
- type: dot_map@100
value: 0.9282346491228068
name: Dot Map@100
---
# SentenceTransformer based on Snowflake/snowflake-arctic-embed-m
This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Snowflake/snowflake-arctic-embed-m](https://huggingface.co/Snowflake/snowflake-arctic-embed-m). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
## Model Details
### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [Snowflake/snowflake-arctic-embed-m](https://huggingface.co/Snowflake/snowflake-arctic-embed-m) <!-- at revision e2b128b9fa60c82b4585512b33e1544224ffff42 -->
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 768 tokens
- **Similarity Function:** Cosine Similarity
<!-- - **Training Dataset:** Unknown -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
### Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("jet-taekyo/snowflake_finetuned_recursive")
# Run inference
sentences = [
'What must lenders provide to consumers who are denied credit under the Fair Credit Reporting Act?',
'that consumers who are denied credit receive "adverse action" notices. Anyone who relies on the information in a \ncredit report to deny a consumer credit must, under the Fair Credit Reporting Act, provide an "adverse action" \nnotice to the consumer, which includes "notice of the reasons a creditor took adverse action on the application \nor on an existing credit account."90 In addition, under the risk-based pricing rule,91 lenders must either inform \nborrowers of their credit score, or else tell consumers when "they are getting worse terms because of \ninformation in their credit report." The CFPB has also asserted that "[t]he law gives every applicant the right to \na specific explanation if their application for credit was denied, and that right is not diminished simply because \na company uses a complex algorithm that it doesn\'t understand."92 Such explanations illustrate a shared value \nthat certain decisions need to be explained.',
'measures to prevent, flag, or take other action in response to outputs that \nreproduce particular training data (e.g., plagiarized, trademarked, patented, \nlicensed content or trade secret material). \nIntellectual Property; CBRN \nInformation or Capabilities',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
## Evaluation
### Metrics
#### Information Retrieval
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| cosine_accuracy@1 | 0.8816 |
| cosine_accuracy@3 | 0.9671 |
| cosine_accuracy@5 | 0.9868 |
| cosine_accuracy@10 | 1.0 |
| cosine_precision@1 | 0.8816 |
| cosine_precision@3 | 0.3224 |
| cosine_precision@5 | 0.1974 |
| cosine_precision@10 | 0.1 |
| cosine_recall@1 | 0.8816 |
| cosine_recall@3 | 0.9671 |
| cosine_recall@5 | 0.9868 |
| cosine_recall@10 | 1.0 |
| cosine_ndcg@10 | 0.946 |
| cosine_mrr@10 | 0.9282 |
| **cosine_map@100** | **0.9282** |
| dot_accuracy@1 | 0.8816 |
| dot_accuracy@3 | 0.9671 |
| dot_accuracy@5 | 0.9868 |
| dot_accuracy@10 | 1.0 |
| dot_precision@1 | 0.8816 |
| dot_precision@3 | 0.3224 |
| dot_precision@5 | 0.1974 |
| dot_precision@10 | 0.1 |
| dot_recall@1 | 0.8816 |
| dot_recall@3 | 0.9671 |
| dot_recall@5 | 0.9868 |
| dot_recall@10 | 1.0 |
| dot_ndcg@10 | 0.946 |
| dot_mrr@10 | 0.9282 |
| dot_map@100 | 0.9282 |
<!--
## Bias, Risks and Limitations
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->
<!--
### Recommendations
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->
## Training Details
### Training Dataset
#### Unnamed Dataset
* Size: 714 training samples
* Columns: <code>sentence_0</code> and <code>sentence_1</code>
* Approximate statistics based on the first 714 samples:
| | sentence_0 | sentence_1 |
|:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|
| type | string | string |
| details | <ul><li>min: 11 tokens</li><li>mean: 18.46 tokens</li><li>max: 32 tokens</li></ul> | <ul><li>min: 21 tokens</li><li>mean: 175.32 tokens</li><li>max: 512 tokens</li></ul> |
* Samples:
| sentence_0 | sentence_1 |
|:---------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| <code>What is the purpose of conducting adversarial testing in the context of GAI risks?</code> | <code>Human-AI Configuration; <br>Information Integrity; Harmful Bias <br>and Homogenization <br>AI Actor Tasks: AI Deployment, Affected Individuals and Communities, End-Users, Operation and Monitoring, TEVV <br> <br>MEASURE 4.2: Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle are <br>informed by input from domain experts and relevant AI Actors to validate whether the system is performing consistently as <br>intended. Results are documented. <br>Action ID <br>Suggested Action <br>GAI Risks <br>MS-4.2-001 <br>Conduct adversarial testing at a regular cadence to map and measure GAI risks, <br>including tests to address attempts to deceive or manipulate the application of <br>provenance techniques or other misuses. Identify vulnerabilities and <br>understand potential misuse scenarios and unintended outputs. <br>Information Integrity; Information <br>Security <br>MS-4.2-002 <br>Evaluate GAI system performance in real-world scenarios to observe its</code> |
| <code>How are measurement results regarding AI system trustworthiness documented and validated?</code> | <code>Human-AI Configuration; <br>Information Integrity; Harmful Bias <br>and Homogenization <br>AI Actor Tasks: AI Deployment, Affected Individuals and Communities, End-Users, Operation and Monitoring, TEVV <br> <br>MEASURE 4.2: Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle are <br>informed by input from domain experts and relevant AI Actors to validate whether the system is performing consistently as <br>intended. Results are documented. <br>Action ID <br>Suggested Action <br>GAI Risks <br>MS-4.2-001 <br>Conduct adversarial testing at a regular cadence to map and measure GAI risks, <br>including tests to address attempts to deceive or manipulate the application of <br>provenance techniques or other misuses. Identify vulnerabilities and <br>understand potential misuse scenarios and unintended outputs. <br>Information Integrity; Information <br>Security <br>MS-4.2-002 <br>Evaluate GAI system performance in real-world scenarios to observe its</code> |
| <code>What types of data provenance information are included in the GAI system inventory entries?</code> | <code>following items in GAI system inventory entries: Data provenance information <br>(e.g., source, signatures, versioning, watermarks); Known issues reported from <br>internal bug tracking or external information sharing resources (e.g., AI incident <br>database, AVID, CVE, NVD, or OECD AI incident monitor); Human oversight roles <br>and responsibilities; Special rights and considerations for intellectual property, <br>licensed works, or personal, privileged, proprietary or sensitive data; Underlying <br>foundation models, versions of underlying models, and access modes. <br>Data Privacy; Human-AI <br>Configuration; Information <br>Integrity; Intellectual Property; <br>Value Chain and Component <br>Integration <br>AI Actor Tasks: Governance and Oversight</code> |
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
```json
{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
768,
512,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1,
1
],
"n_dims_per_step": -1
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `eval_strategy`: steps
- `per_device_train_batch_size`: 20
- `per_device_eval_batch_size`: 20
- `num_train_epochs`: 5
- `multi_dataset_batch_sampler`: round_robin
#### All Hyperparameters
<details><summary>Click to expand</summary>
- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: steps
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 20
- `per_device_eval_batch_size`: 20
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 1
- `eval_accumulation_steps`: None
- `torch_empty_cache_steps`: None
- `learning_rate`: 5e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1
- `num_train_epochs`: 5
- `max_steps`: -1
- `lr_scheduler_type`: linear
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.0
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: False
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: False
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: False
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`:
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: False
- `eval_use_gather_object`: False
- `batch_sampler`: batch_sampler
- `multi_dataset_batch_sampler`: round_robin
</details>
### Training Logs
| Epoch | Step | cosine_map@100 |
|:------:|:----:|:--------------:|
| 1.0 | 36 | 0.9145 |
| 1.3889 | 50 | 0.9256 |
| 2.0 | 72 | 0.9246 |
| 2.7778 | 100 | 0.9282 |
### Framework Versions
- Python: 3.11.9
- Sentence Transformers: 3.1.0
- Transformers: 4.44.2
- PyTorch: 2.4.1+cu121
- Accelerate: 0.34.2
- Datasets: 3.0.0
- Tokenizers: 0.19.1
## Citation
### BibTeX
#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
```
#### MatryoshkaLoss
```bibtex
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```
#### MultipleNegativesRankingLoss
```bibtex
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->