Edit model card

SentenceTransformer based on jeffwan/mmarco-mMiniLMv2-L12-H384-v1

This is a sentence-transformers model finetuned from jeffwan/mmarco-mMiniLMv2-L12-H384-v1. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: jeffwan/mmarco-mMiniLMv2-L12-H384-v1
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 384 tokens
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("adriansanz/sitges10242608-4ep-rerank")
# Run inference
sentences = [
    "Aquest tràmit permet sol·licitar la llicència per a realitzar obres d'excavació a la via pública per a la instal·lació o reparació d'infraestructures de serveis i subministraments.",
    'Quin és el paper de la via pública en aquest tràmit?',
    "Quin és l'objectiu de presentar una denúncia per presumpta infracció urbanística?",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.0366
cosine_accuracy@3 0.0797
cosine_accuracy@5 0.1121
cosine_accuracy@10 0.1832
cosine_precision@1 0.0366
cosine_precision@3 0.0266
cosine_precision@5 0.0224
cosine_precision@10 0.0183
cosine_recall@1 0.0366
cosine_recall@3 0.0797
cosine_recall@5 0.1121
cosine_recall@10 0.1832
cosine_ndcg@10 0.0978
cosine_mrr@10 0.0721
cosine_map@100 0.0851

Information Retrieval

Metric Value
cosine_accuracy@1 0.0366
cosine_accuracy@3 0.0797
cosine_accuracy@5 0.1121
cosine_accuracy@10 0.1832
cosine_precision@1 0.0366
cosine_precision@3 0.0266
cosine_precision@5 0.0224
cosine_precision@10 0.0183
cosine_recall@1 0.0366
cosine_recall@3 0.0797
cosine_recall@5 0.1121
cosine_recall@10 0.1832
cosine_ndcg@10 0.0978
cosine_mrr@10 0.0721
cosine_map@100 0.0851

Information Retrieval

Metric Value
cosine_accuracy@1 0.0388
cosine_accuracy@3 0.0862
cosine_accuracy@5 0.1228
cosine_accuracy@10 0.2091
cosine_precision@1 0.0388
cosine_precision@3 0.0287
cosine_precision@5 0.0246
cosine_precision@10 0.0209
cosine_recall@1 0.0388
cosine_recall@3 0.0862
cosine_recall@5 0.1228
cosine_recall@10 0.2091
cosine_ndcg@10 0.1089
cosine_mrr@10 0.0789
cosine_map@100 0.0926

Information Retrieval

Metric Value
cosine_accuracy@1 0.0409
cosine_accuracy@3 0.0884
cosine_accuracy@5 0.1164
cosine_accuracy@10 0.1961
cosine_precision@1 0.0409
cosine_precision@3 0.0295
cosine_precision@5 0.0233
cosine_precision@10 0.0196
cosine_recall@1 0.0409
cosine_recall@3 0.0884
cosine_recall@5 0.1164
cosine_recall@10 0.1961
cosine_ndcg@10 0.1053
cosine_mrr@10 0.0782
cosine_map@100 0.0931

Information Retrieval

Metric Value
cosine_accuracy@1 0.0409
cosine_accuracy@3 0.0905
cosine_accuracy@5 0.1121
cosine_accuracy@10 0.1832
cosine_precision@1 0.0409
cosine_precision@3 0.0302
cosine_precision@5 0.0224
cosine_precision@10 0.0183
cosine_recall@1 0.0409
cosine_recall@3 0.0905
cosine_recall@5 0.1121
cosine_recall@10 0.1832
cosine_ndcg@10 0.1001
cosine_mrr@10 0.0751
cosine_map@100 0.09

Training Details

Training Dataset

Unnamed Dataset

  • Size: 4,173 training samples
  • Columns: positive and anchor
  • Approximate statistics based on the first 1000 samples:
    positive anchor
    type string string
    details
    • min: 9 tokens
    • mean: 49.38 tokens
    • max: 190 tokens
    • min: 10 tokens
    • mean: 21.0 tokens
    • max: 48 tokens
  • Samples:
    positive anchor
    Havent-se d'acreditar la matriculació i inscripció en el respectiu centre públic o concertat, així com el cost de les llars d'infants, de l'educació especialitzada per les discapacitats físiques, psíquiques i sensorials en centres públics, concertats o privats. Quin és el requisit per acreditar la llar d'infants?
    El volant històric de convivència és el document que informa de la residencia en el municipi de Sitges, així com altres fets relatius a l'empadronament d'una persona, i detalla tots els domicilis, la data inicial i final en els que ha estat empadronada en cadascun d'ells, i les persones amb les què constava inscrites, segons les dades que consten al Padró Municipal d'Habitants fins a la data d'expedició. Quin és el propòsit del volant històric de convivència?
    Instal·lació de tanques sense obra. Quins són els exemples d'instal·lacions que es poden comunicar amb aquest tràmit?
  • Loss: MatryoshkaLoss with these parameters:
    {
        "loss": "MultipleNegativesRankingLoss",
        "matryoshka_dims": [
            384,
            256,
            128,
            64
        ],
        "matryoshka_weights": [
            1,
            1,
            1,
            1
        ],
        "n_dims_per_step": -1
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: epoch
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • gradient_accumulation_steps: 16
  • num_train_epochs: 5
  • lr_scheduler_type: cosine
  • warmup_ratio: 0.2
  • bf16: True
  • tf32: False
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: epoch
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 16
  • eval_accumulation_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 5
  • max_steps: -1
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.2
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: True
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: False
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss dim_128_cosine_map@100 dim_256_cosine_map@100 dim_512_cosine_map@100 dim_64_cosine_map@100 dim_768_cosine_map@100
0.6130 10 11.7801 - - - - -
0.9808 16 - 0.0132 0.0103 0.0105 0.0116 0.0105
1.2261 20 10.5594 - - - - -
1.8391 30 9.0859 - - - - -
1.9617 32 - 0.0337 0.0302 0.0298 0.0323 0.0298
2.4521 40 7.5747 - - - - -
2.9425 48 - 0.0811 0.0765 0.0679 0.0742 0.0679
3.0651 50 5.7656 - - - - -
3.6782 60 4.7495 - - - - -
3.9847 65 - 0.0926 0.0929 0.0822 0.0886 0.0822
4.2912 70 4.1026 - - - - -
4.9042 80 3.8201 0.0931 0.0926 0.0851 0.09 0.0851
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.0.1
  • Transformers: 4.42.4
  • PyTorch: 2.4.0+cu121
  • Accelerate: 0.34.0.dev0
  • Datasets: 2.21.0
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
3
Safetensors
Model size
118M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for adriansanz/sitges10242608-4ep-rerank

Finetuned
(1)
this model

Evaluation results