Hugging Face's logo

mrzaizai2k
/
distilroberta-base-sentence-transformer-triplets

Sentence Similarity
sentence-transformers
Safetensors
distilbert
feature-extraction
Generated from Trainer
dataset_size:101762
loss:TripletLoss
text-embeddings-inference
Inference Endpoints
Model card Files Community
distilroberta-base-sentence-transformer-triplets / README.md
Bảo Mai Chí
Add new SentenceTransformer model.
1938e4a verified
preview code
|
raw
history blame contribute delete
13 kB
metadata
language: []
library_name: sentence-transformers
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:101762
  - loss:TripletLoss
base_model: sentence-transformers/distiluse-base-multilingual-cased-v2
datasets: []
widget:
  - source_sentence: How do I clean the screen of my Toshiba TV?
    sentences:
      - How can I clear screen overlay from my Samsung Galaxy 6?
      - Why do police forces exist?
      - What is the best way to clean a flat screen monitor?
  - source_sentence: What was the first video you watched on YouTube?
    sentences:
      - What was the first Youtube video you ever watched?
      - What was the first music video ever produced?
      - >-
        What was the long term effect of Hitler's desire to exterminate the
        Jewish people?
  - source_sentence: What should I do to recover my data from a hard disk?
    sentences:
      - How do I recover my deleted data files from a hard disk?
      - What's the best Linux operating System distro for beginners?
      - >-
        Formated Data Recovery – Recover Data from Memory Card, Disk Drive, USB,
        External Drive?
  - source_sentence: What are your personal top ten music albums of all time?
    sentences:
      - What are your top 10 favourite songs of all time?
      - What are the Top 10 music albums of all time on your list?
      - >-
        What stream should I take in 11th if I have to become an automobile
        engineer?
  - source_sentence: What is the best website to learn coding independently?
    sentences:
      - >-
        What are some of the best website to learn programming from being a
        total beginner?
      - What books do I need to read to learn more about Sufism?
      - >-
        What is the best (and fastest) way to learn how to code (web
        development)?
pipeline_tag: sentence-similarity

SentenceTransformer based on sentence-transformers/distiluse-base-multilingual-cased-v2

This is a sentence-transformers model finetuned from sentence-transformers/distiluse-base-multilingual-cased-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: DistilBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("chibao24/distilroberta-base-sentence-transformer-triplets")
# Run inference
sentences = [
    'What is the best website to learn coding independently?',
    'What are some of the best website to learn programming from being a total beginner?',
    'What is the best (and fastest) way to learn how to code (web development)?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

  • Size: 101,762 training samples
  • Columns: sentence_0, sentence_1, and sentence_2
  • Approximate statistics based on the first 1000 samples:
    sentence_0 sentence_1 sentence_2
    type string string string
    details
    • min: 6 tokens
    • mean: 14.7 tokens
    • max: 46 tokens
    • min: 6 tokens
    • mean: 14.66 tokens
    • max: 41 tokens
    • min: 6 tokens
    • mean: 15.22 tokens
    • max: 84 tokens
  • Samples:
    sentence_0 sentence_1 sentence_2
    What are the differences between "be made of" and "be made from"? What's the difference between "made of" and "made from"? What is the difference between make and craft?
    How can we use the word "inertia" in a sentence? How can the word "inertia" be used in a sentence? What is inertia actually?
    Who are the new (i.e. first-time) Top Question Writers for 2017? Who are the top question writers for 2017? Who are the 2016 Top Writers?
  • Loss: TripletLoss with these parameters:
    {
    "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
    "triplet_margin": 5
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 128
  • per_device_eval_batch_size: 128
  • num_train_epochs: 4
  • multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: no
  • prediction_loss_only: True
  • per_device_train_batch_size: 128
  • per_device_eval_batch_size: 128
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1
  • num_train_epochs: 4
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: round_robin

Training Logs

Epoch Step Training Loss
0.6281 500 4.2255
1.2563 1000 3.484
1.8844 1500 2.8611
2.5126 2000 2.4607
3.1407 2500 2.148
3.7688 3000 1.8583

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.0.1
  • Transformers: 4.41.2
  • PyTorch: 2.1.2+cu121
  • Accelerate: 0.32.1
  • Datasets: 2.19.0
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss 

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification}, 
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}