SentenceTransformer based on intfloat/multilingual-e5-small
This is a sentence-transformers model finetuned from intfloat/multilingual-e5-small. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: intfloat/multilingual-e5-small
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 384 tokens
- Similarity Function: Cosine Similarity
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("srikarvar/fine_tuned_model_15")
# Run inference
sentences = [
'Who wrote the book "To Kill a Mockingbird"?',
'Who wrote the book "1984"?',
'At what speed does light travel?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
Evaluation
Metrics
Binary Classification
- Dataset:
pair-class-dev
- Evaluated with
BinaryClassificationEvaluator
Metric | Value |
---|---|
cosine_accuracy | 0.8768 |
cosine_accuracy_threshold | 0.8267 |
cosine_f1 | 0.897 |
cosine_f1_threshold | 0.8267 |
cosine_precision | 0.881 |
cosine_recall | 0.9136 |
cosine_ap | 0.9301 |
dot_accuracy | 0.8768 |
dot_accuracy_threshold | 0.8267 |
dot_f1 | 0.897 |
dot_f1_threshold | 0.8267 |
dot_precision | 0.881 |
dot_recall | 0.9136 |
dot_ap | 0.9301 |
manhattan_accuracy | 0.8732 |
manhattan_accuracy_threshold | 8.953 |
manhattan_f1 | 0.893 |
manhattan_f1_threshold | 9.028 |
manhattan_precision | 0.8848 |
manhattan_recall | 0.9012 |
manhattan_ap | 0.9285 |
euclidean_accuracy | 0.8768 |
euclidean_accuracy_threshold | 0.5886 |
euclidean_f1 | 0.897 |
euclidean_f1_threshold | 0.5886 |
euclidean_precision | 0.881 |
euclidean_recall | 0.9136 |
euclidean_ap | 0.9301 |
max_accuracy | 0.8768 |
max_accuracy_threshold | 8.953 |
max_f1 | 0.897 |
max_f1_threshold | 9.028 |
max_precision | 0.8848 |
max_recall | 0.9136 |
max_ap | 0.9301 |
Binary Classification
- Dataset:
pair-class-test
- Evaluated with
BinaryClassificationEvaluator
Metric | Value |
---|---|
cosine_accuracy | 0.8768 |
cosine_accuracy_threshold | 0.8267 |
cosine_f1 | 0.897 |
cosine_f1_threshold | 0.8267 |
cosine_precision | 0.881 |
cosine_recall | 0.9136 |
cosine_ap | 0.9301 |
dot_accuracy | 0.8768 |
dot_accuracy_threshold | 0.8267 |
dot_f1 | 0.897 |
dot_f1_threshold | 0.8267 |
dot_precision | 0.881 |
dot_recall | 0.9136 |
dot_ap | 0.9301 |
manhattan_accuracy | 0.8732 |
manhattan_accuracy_threshold | 8.953 |
manhattan_f1 | 0.893 |
manhattan_f1_threshold | 9.028 |
manhattan_precision | 0.8848 |
manhattan_recall | 0.9012 |
manhattan_ap | 0.9285 |
euclidean_accuracy | 0.8768 |
euclidean_accuracy_threshold | 0.5886 |
euclidean_f1 | 0.897 |
euclidean_f1_threshold | 0.5886 |
euclidean_precision | 0.881 |
euclidean_recall | 0.9136 |
euclidean_ap | 0.9301 |
max_accuracy | 0.8768 |
max_accuracy_threshold | 8.953 |
max_f1 | 0.897 |
max_f1_threshold | 9.028 |
max_precision | 0.8848 |
max_recall | 0.9136 |
max_ap | 0.9301 |
Training Details
Training Dataset
Unnamed Dataset
- Size: 2,476 training samples
- Columns:
label
,sentence1
, andsentence2
- Approximate statistics based on the first 1000 samples:
label sentence1 sentence2 type int string string details - 0: ~40.20%
- 1: ~59.80%
- min: 6 tokens
- mean: 16.35 tokens
- max: 98 tokens
- min: 4 tokens
- mean: 16.06 tokens
- max: 98 tokens
- Samples:
label sentence1 sentence2 1
The ImageNet dataset is used for training models to classify images into various categories.
A model is trained using the ImageNet dataset to classify images into distinct categories.
1
No, it doesn't exist in version 5.3.1.
Version 5.3.1 does not contain it.
0
Can you help me with my homework?
Can you do my homework for me?
- Loss:
OnlineContrastiveLoss
Evaluation Dataset
Unnamed Dataset
- Size: 276 evaluation samples
- Columns:
label
,sentence1
, andsentence2
- Approximate statistics based on the first 276 samples:
label sentence1 sentence2 type int string string details - 0: ~41.30%
- 1: ~58.70%
- min: 6 tokens
- mean: 15.56 tokens
- max: 87 tokens
- min: 5 tokens
- mean: 15.34 tokens
- max: 86 tokens
- Samples:
label sentence1 sentence2 0
What are the challenges of AI in cybersecurity?
How is AI used to enhance cybersecurity?
1
You can find the SYSTEM log documentation on the main version. Click on the provided link to redirect to the main version of the documentation.
The SYSTEM log documentation can be accessed by clicking on the link which will take you to the main version.
1
What is the capital of Italy?
Name the capital city of Italy
- Loss:
OnlineContrastiveLoss
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy
: epochper_device_train_batch_size
: 32per_device_eval_batch_size
: 32gradient_accumulation_steps
: 2num_train_epochs
: 4warmup_ratio
: 0.1load_best_model_at_end
: Trueoptim
: adamw_torch_fusedbatch_sampler
: no_duplicates
All Hyperparameters
Click to expand
overwrite_output_dir
: Falsedo_predict
: Falseeval_strategy
: epochprediction_loss_only
: Trueper_device_train_batch_size
: 32per_device_eval_batch_size
: 32per_gpu_train_batch_size
: Noneper_gpu_eval_batch_size
: Nonegradient_accumulation_steps
: 2eval_accumulation_steps
: Nonelearning_rate
: 5e-05weight_decay
: 0.0adam_beta1
: 0.9adam_beta2
: 0.999adam_epsilon
: 1e-08max_grad_norm
: 1.0num_train_epochs
: 4max_steps
: -1lr_scheduler_type
: linearlr_scheduler_kwargs
: {}warmup_ratio
: 0.1warmup_steps
: 0log_level
: passivelog_level_replica
: warninglog_on_each_node
: Truelogging_nan_inf_filter
: Truesave_safetensors
: Truesave_on_each_node
: Falsesave_only_model
: Falserestore_callback_states_from_checkpoint
: Falseno_cuda
: Falseuse_cpu
: Falseuse_mps_device
: Falseseed
: 42data_seed
: Nonejit_mode_eval
: Falseuse_ipex
: Falsebf16
: Falsefp16
: Falsefp16_opt_level
: O1half_precision_backend
: autobf16_full_eval
: Falsefp16_full_eval
: Falsetf32
: Nonelocal_rank
: 0ddp_backend
: Nonetpu_num_cores
: Nonetpu_metrics_debug
: Falsedebug
: []dataloader_drop_last
: Falsedataloader_num_workers
: 0dataloader_prefetch_factor
: Nonepast_index
: -1disable_tqdm
: Falseremove_unused_columns
: Truelabel_names
: Noneload_best_model_at_end
: Trueignore_data_skip
: Falsefsdp
: []fsdp_min_num_params
: 0fsdp_config
: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap
: Noneaccelerator_config
: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed
: Nonelabel_smoothing_factor
: 0.0optim
: adamw_torch_fusedoptim_args
: Noneadafactor
: Falsegroup_by_length
: Falselength_column_name
: lengthddp_find_unused_parameters
: Noneddp_bucket_cap_mb
: Noneddp_broadcast_buffers
: Falsedataloader_pin_memory
: Truedataloader_persistent_workers
: Falseskip_memory_metrics
: Trueuse_legacy_prediction_loop
: Falsepush_to_hub
: Falseresume_from_checkpoint
: Nonehub_model_id
: Nonehub_strategy
: every_savehub_private_repo
: Falsehub_always_push
: Falsegradient_checkpointing
: Falsegradient_checkpointing_kwargs
: Noneinclude_inputs_for_metrics
: Falseeval_do_concat_batches
: Truefp16_backend
: autopush_to_hub_model_id
: Nonepush_to_hub_organization
: Nonemp_parameters
:auto_find_batch_size
: Falsefull_determinism
: Falsetorchdynamo
: Noneray_scope
: lastddp_timeout
: 1800torch_compile
: Falsetorch_compile_backend
: Nonetorch_compile_mode
: Nonedispatch_batches
: Nonesplit_batches
: Noneinclude_tokens_per_second
: Falseinclude_num_input_tokens_seen
: Falseneftune_noise_alpha
: Noneoptim_target_modules
: Nonebatch_eval_metrics
: Falsebatch_sampler
: no_duplicatesmulti_dataset_batch_sampler
: proportional
Training Logs
Epoch | Step | Training Loss | loss | pair-class-dev_max_ap | pair-class-test_max_ap |
---|---|---|---|---|---|
0 | 0 | - | - | 0.7876 | - |
0.2564 | 10 | 1.5794 | - | - | - |
0.5128 | 20 | 0.8392 | - | - | - |
0.7692 | 30 | 0.7812 | - | - | - |
1.0 | 39 | - | 0.8081 | 0.9138 | - |
1.0256 | 40 | 0.6505 | - | - | - |
1.2821 | 50 | 0.57 | - | - | - |
1.5385 | 60 | 0.3015 | - | - | - |
1.7949 | 70 | 0.3091 | - | - | - |
2.0 | 78 | - | 0.7483 | 0.9267 | - |
2.0513 | 80 | 0.3988 | - | - | - |
2.3077 | 90 | 0.1801 | - | - | - |
2.5641 | 100 | 0.1166 | - | - | - |
2.8205 | 110 | 0.1255 | - | - | - |
3.0 | 117 | - | 0.7106 | 0.9284 | - |
3.0769 | 120 | 0.2034 | - | - | - |
3.3333 | 130 | 0.0329 | - | - | - |
3.5897 | 140 | 0.0805 | - | - | - |
3.8462 | 150 | 0.0816 | - | - | - |
4.0 | 156 | - | 0.6969 | 0.9301 | 0.9301 |
- The bold row denotes the saved checkpoint.
Framework Versions
- Python: 3.10.12
- Sentence Transformers: 3.1.0
- Transformers: 4.41.2
- PyTorch: 2.1.2+cu121
- Accelerate: 0.34.2
- Datasets: 2.19.1
- Tokenizers: 0.19.1
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
- Downloads last month
- 1
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for srikarvar/fine_tuned_model_15
Base model
intfloat/multilingual-e5-smallEvaluation results
- Cosine Accuracy on pair class devself-reported0.877
- Cosine Accuracy Threshold on pair class devself-reported0.827
- Cosine F1 on pair class devself-reported0.897
- Cosine F1 Threshold on pair class devself-reported0.827
- Cosine Precision on pair class devself-reported0.881
- Cosine Recall on pair class devself-reported0.914
- Cosine Ap on pair class devself-reported0.930
- Dot Accuracy on pair class devself-reported0.877
- Dot Accuracy Threshold on pair class devself-reported0.827
- Dot F1 on pair class devself-reported0.897