MPNet base trained on AllNLI triplets

This is a sentence-transformers model finetuned from microsoft/mpnet-base on the all-nli dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: microsoft/mpnet-base
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 tokens
  • Similarity Function: Cosine Similarity
  • Training Dataset:
  • Language: en
  • License: apache-2.0

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("chanbistec/mpnet-base-all-nli-triplet")
# Run inference
sentences = [
    'A construction worker peeking out of a manhole while his coworker sits on the sidewalk smiling.',
    'A worker is looking out of a manhole.',
    'The workers are both inside the manhole.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

Metric Value
cosine_accuracy 0.9142
dot_accuracy 0.0844
manhattan_accuracy 0.9098
euclidean_accuracy 0.9098
max_accuracy 0.9142

Triplet

Metric Value
cosine_accuracy 0.9265
dot_accuracy 0.0735
manhattan_accuracy 0.9187
euclidean_accuracy 0.918
max_accuracy 0.9265

Training Details

Training Dataset

all-nli

  • Dataset: all-nli at d482672
  • Size: 557,850 training samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 7 tokens
    • mean: 10.46 tokens
    • max: 46 tokens
    • min: 6 tokens
    • mean: 12.81 tokens
    • max: 40 tokens
    • min: 5 tokens
    • mean: 13.4 tokens
    • max: 50 tokens
  • Samples:
    anchor positive negative
    A person on a horse jumps over a broken down airplane. A person is outdoors, on a horse. A person is at a diner, ordering an omelette.
    Children smiling and waving at camera There are children present The kids are frowning
    A boy is jumping on skateboard in the middle of a red bridge. The boy does a skateboarding trick. The boy skates down the sidewalk.
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim"
    }
    

Evaluation Dataset

all-nli

  • Dataset: all-nli at d482672
  • Size: 6,584 evaluation samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 6 tokens
    • mean: 17.95 tokens
    • max: 63 tokens
    • min: 4 tokens
    • mean: 9.78 tokens
    • max: 29 tokens
    • min: 5 tokens
    • mean: 10.35 tokens
    • max: 29 tokens
  • Samples:
    anchor positive negative
    Two women are embracing while holding to go packages. Two woman are holding packages. The men are fighting outside a deli.
    Two young children in blue jerseys, one with the number 9 and one with the number 2 are standing on wooden steps in a bathroom and washing their hands in a sink. Two kids in numbered jerseys wash their hands. Two kids in jackets walk to school.
    A man selling donuts to a customer during a world exhibition event held in the city of Angeles A man selling donuts to a customer. A woman drinks her coffee in a small cafe.
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim"
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • learning_rate: 2e-05
  • num_train_epochs: 1
  • warmup_ratio: 0.1
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 1
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • eval_use_gather_object: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss loss all-nli-dev_max_accuracy all-nli-test_max_accuracy
0 0 - - 0.6832 -
0.016 100 3.0282 1.5782 0.7752 -
0.032 200 1.2529 0.9154 0.7991 -
0.048 300 1.4472 0.7901 0.8103 -
0.064 400 0.9059 0.7468 0.8114 -
0.08 500 0.8663 0.8423 0.7981 -
0.096 600 1.0836 0.8995 0.8010 -
0.112 700 0.9315 0.8971 0.8100 -
0.128 800 1.1273 0.9654 0.8012 -
0.144 900 1.1194 0.9318 0.8303 -
0.16 1000 1.0911 0.9048 0.8038 -
0.176 1100 1.1332 0.9340 0.8039 -
0.192 1200 1.0154 0.9041 0.8076 -
0.208 1300 0.7995 0.9301 0.7959 -
0.224 1400 0.7614 0.8275 0.8071 -
0.24 1500 0.8724 0.7973 0.8173 -
0.256 1600 0.6751 0.7916 0.8197 -
0.272 1700 0.8933 0.8572 0.8194 -
0.288 1800 0.8585 0.8560 0.8056 -
0.304 1900 0.8354 0.7987 0.8123 -
0.32 2000 0.7484 0.7559 0.8348 -
0.336 2100 0.6047 0.7532 0.8471 -
0.352 2200 0.6221 0.6956 0.8665 -
0.368 2300 0.8332 0.7214 0.8542 -
0.384 2400 0.7755 0.7007 0.8481 -
0.4 2500 0.6912 0.7505 0.8499 -
0.416 2600 0.6169 0.6536 0.8591 -
0.432 2700 0.8907 0.7240 0.8560 -
0.448 2800 0.8576 0.6790 0.8499 -
0.464 2900 0.8057 0.6870 0.8575 -
0.48 3000 0.6928 0.6540 0.8641 -
0.496 3100 0.7566 0.6419 0.8682 -
0.512 3200 0.5757 0.6109 0.8783 -
0.528 3300 0.601 0.5481 0.8914 -
0.544 3400 0.5105 0.5853 0.8820 -
0.56 3500 0.5116 0.5918 0.8961 -
0.576 3600 0.495 0.5546 0.8897 -
0.592 3700 0.5585 0.5457 0.8970 -
0.608 3800 0.4778 0.5056 0.9020 -
0.624 3900 0.5116 0.5203 0.9019 -
0.64 4000 0.753 0.5490 0.9019 -
0.656 4100 0.9207 0.5447 0.9049 -
0.672 4200 0.8695 0.4996 0.9055 -
0.688 4300 0.6867 0.4825 0.9107 -
0.704 4400 0.5961 0.4670 0.9166 -
0.72 4500 0.5547 0.4748 0.9104 -
0.736 4600 0.6145 0.4636 0.9145 -
0.752 4700 0.6643 0.4806 0.9128 -
0.768 4800 0.6134 0.4521 0.9110 -
0.784 4900 0.5847 0.4627 0.9080 -
0.8 5000 0.6482 0.4853 0.9107 -
0.816 5100 0.5103 0.4374 0.9104 -
0.832 5200 0.5639 0.4306 0.9089 -
0.848 5300 0.5247 0.4418 0.9116 -
0.864 5400 0.6094 0.4564 0.9101 -
0.88 5500 0.5296 0.4394 0.9092 -
0.896 5600 0.5469 0.4316 0.9101 -
0.912 5700 0.6061 0.4258 0.9124 -
0.928 5800 0.5456 0.4167 0.9113 -
0.944 5900 0.6776 0.4168 0.9108 -
0.96 6000 0.7401 0.4267 0.9139 -
0.976 6100 0.6568 0.4227 0.9140 -
0.992 6200 0.0002 0.4224 0.9142 -
1.0 6250 - - - 0.9265

Framework Versions

  • Python: 3.12.4
  • Sentence Transformers: 3.1.0
  • Transformers: 4.44.2
  • PyTorch: 2.4.1
  • Accelerate: 0.34.2
  • Datasets: 3.0.0
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
27
Safetensors
Model size
109M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for chanbistec/mpnet-base-all-nli-triplet

Finetuned
(47)
this model

Dataset used to train chanbistec/mpnet-base-all-nli-triplet

Evaluation results