SentenceTransformer based on answerdotai/ModernBERT-base
This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the csv dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: answerdotai/ModernBERT-base
- Maximum Sequence Length: 8192 tokens
- Output Dimensionality: 768 dimensions
- Similarity Function: Cosine Similarity
- Training Dataset:
- csv
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
'what is the connection between cancer and the cell cycle?',
'Conclusion. Cancer is unchecked cell growth. Mutations in genes can cause cancer by accelerating cell division rates or inhibiting normal controls on the system, such as cell cycle arrest or programmed cell death. As a mass of cancerous cells grows, it can develop into a tumor.',
'Biology',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
Evaluation
Metrics
Information Retrieval
- Datasets:
NanoNQ
andNanoMSMARCO
- Evaluated with
InformationRetrievalEvaluator
Metric | NanoNQ | NanoMSMARCO |
---|---|---|
cosine_accuracy@1 | 0.1 | 0.12 |
cosine_accuracy@3 | 0.18 | 0.28 |
cosine_accuracy@5 | 0.24 | 0.4 |
cosine_accuracy@10 | 0.34 | 0.52 |
cosine_precision@1 | 0.1 | 0.12 |
cosine_precision@3 | 0.06 | 0.0933 |
cosine_precision@5 | 0.048 | 0.08 |
cosine_precision@10 | 0.034 | 0.052 |
cosine_recall@1 | 0.1 | 0.12 |
cosine_recall@3 | 0.15 | 0.28 |
cosine_recall@5 | 0.21 | 0.4 |
cosine_recall@10 | 0.31 | 0.52 |
cosine_ndcg@10 | 0.1934 | 0.2985 |
cosine_mrr@10 | 0.1659 | 0.2304 |
cosine_map@100 | 0.1764 | 0.2469 |
Nano BEIR
- Dataset:
NanoBEIR_mean
- Evaluated with
NanoBEIREvaluator
Metric | Value |
---|---|
cosine_accuracy@1 | 0.11 |
cosine_accuracy@3 | 0.23 |
cosine_accuracy@5 | 0.32 |
cosine_accuracy@10 | 0.43 |
cosine_precision@1 | 0.11 |
cosine_precision@3 | 0.0767 |
cosine_precision@5 | 0.064 |
cosine_precision@10 | 0.043 |
cosine_recall@1 | 0.11 |
cosine_recall@3 | 0.215 |
cosine_recall@5 | 0.305 |
cosine_recall@10 | 0.415 |
cosine_ndcg@10 | 0.246 |
cosine_mrr@10 | 0.1982 |
cosine_map@100 | 0.2117 |
Training Details
Training Dataset
csv
- Dataset: csv
- Size: 100,006 training samples
- Columns:
question
,answer
, andcategory
- Approximate statistics based on the first 1000 samples:
question answer category type string string string details - min: 8 tokens
- mean: 11.91 tokens
- max: 20 tokens
- min: 16 tokens
- mean: 57.49 tokens
- max: 136 tokens
- min: 4 tokens
- mean: 4.0 tokens
- max: 4 tokens
- Samples:
question answer category how many times a week should you use heat on your hair?
Don't style hair with heat every day. Hot tools can also make hair look crispy and create split ends if overused. Blow out hair 3-5 times a week and try to limit your flat iron/curling iron usage to 1-2 times a week.”
Medicine
do african violets like to be root bound?
African violets only bloom when they're root bound. When it is time to repot, be sure to use an organic potting soil made specifically for African violets, such as Espoma's African Violet Mix. They flower best in small pots — choose one that's about a third of the diameter of their leaf spread.
Biology
is pgwp exempt from lmia?
The PGWP is exempt from Labour Market Impact Assessment (LMIA) requirements. The candidate must have attended a recognized post-secondary school, or a secondary school that offers qualifying programs, for at least eight months.
Medicine
- Loss:
CachedMultipleNegativesRankingLoss
with these parameters:{ "scale": 20.0, "similarity_fct": "cos_sim" }
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy
: stepsper_device_train_batch_size
: 256per_device_eval_batch_size
: 256learning_rate
: 0.0001num_train_epochs
: 1warmup_ratio
: 0.05bf16
: Truebatch_sampler
: no_duplicates
All Hyperparameters
Click to expand
overwrite_output_dir
: Falsedo_predict
: Falseeval_strategy
: stepsprediction_loss_only
: Trueper_device_train_batch_size
: 256per_device_eval_batch_size
: 256per_gpu_train_batch_size
: Noneper_gpu_eval_batch_size
: Nonegradient_accumulation_steps
: 1eval_accumulation_steps
: Nonetorch_empty_cache_steps
: Nonelearning_rate
: 0.0001weight_decay
: 0.0adam_beta1
: 0.9adam_beta2
: 0.999adam_epsilon
: 1e-08max_grad_norm
: 1.0num_train_epochs
: 1max_steps
: -1lr_scheduler_type
: linearlr_scheduler_kwargs
: {}warmup_ratio
: 0.05warmup_steps
: 0log_level
: passivelog_level_replica
: warninglog_on_each_node
: Truelogging_nan_inf_filter
: Truesave_safetensors
: Truesave_on_each_node
: Falsesave_only_model
: Falserestore_callback_states_from_checkpoint
: Falseno_cuda
: Falseuse_cpu
: Falseuse_mps_device
: Falseseed
: 42data_seed
: Nonejit_mode_eval
: Falseuse_ipex
: Falsebf16
: Truefp16
: Falsefp16_opt_level
: O1half_precision_backend
: autobf16_full_eval
: Falsefp16_full_eval
: Falsetf32
: Nonelocal_rank
: 0ddp_backend
: Nonetpu_num_cores
: Nonetpu_metrics_debug
: Falsedebug
: []dataloader_drop_last
: Falsedataloader_num_workers
: 0dataloader_prefetch_factor
: Nonepast_index
: -1disable_tqdm
: Falseremove_unused_columns
: Truelabel_names
: Noneload_best_model_at_end
: Falseignore_data_skip
: Falsefsdp
: []fsdp_min_num_params
: 0fsdp_config
: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap
: Noneaccelerator_config
: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed
: Nonelabel_smoothing_factor
: 0.0optim
: adamw_torchoptim_args
: Noneadafactor
: Falsegroup_by_length
: Falselength_column_name
: lengthddp_find_unused_parameters
: Noneddp_bucket_cap_mb
: Noneddp_broadcast_buffers
: Falsedataloader_pin_memory
: Truedataloader_persistent_workers
: Falseskip_memory_metrics
: Trueuse_legacy_prediction_loop
: Falsepush_to_hub
: Falseresume_from_checkpoint
: Nonehub_model_id
: Nonehub_strategy
: every_savehub_private_repo
: Nonehub_always_push
: Falsegradient_checkpointing
: Falsegradient_checkpointing_kwargs
: Noneinclude_inputs_for_metrics
: Falseinclude_for_metrics
: []eval_do_concat_batches
: Truefp16_backend
: autopush_to_hub_model_id
: Nonepush_to_hub_organization
: Nonemp_parameters
:auto_find_batch_size
: Falsefull_determinism
: Falsetorchdynamo
: Noneray_scope
: lastddp_timeout
: 1800torch_compile
: Falsetorch_compile_backend
: Nonetorch_compile_mode
: Nonedispatch_batches
: Nonesplit_batches
: Noneinclude_tokens_per_second
: Falseinclude_num_input_tokens_seen
: Falseneftune_noise_alpha
: Noneoptim_target_modules
: Nonebatch_eval_metrics
: Falseeval_on_start
: Falseuse_liger_kernel
: Falseeval_use_gather_object
: Falseaverage_tokens_across_devices
: Falseprompts
: Nonebatch_sampler
: no_duplicatesmulti_dataset_batch_sampler
: proportional
Training Logs
Epoch | Step | Training Loss | NanoNQ_cosine_ndcg@10 | NanoMSMARCO_cosine_ndcg@10 | NanoBEIR_mean_cosine_ndcg@10 |
---|---|---|---|---|---|
0 | 0 | - | 0.0388 | 0.0863 | 0.0626 |
0.0763 | 10 | 0.5482 | - | - | - |
0.1527 | 20 | 0.1079 | - | - | - |
0.2290 | 30 | 0.1491 | - | - | - |
0.3053 | 40 | 0.1381 | - | - | - |
0.3817 | 50 | 0.0873 | 0.0909 | 0.2197 | 0.1553 |
0.4580 | 60 | 0.133 | - | - | - |
0.5344 | 70 | 0.0539 | - | - | - |
0.6107 | 80 | 0.029 | - | - | - |
0.6870 | 90 | 0.0008 | - | - | - |
0.7634 | 100 | 0.0997 | 0.1982 | 0.2657 | 0.2320 |
0.8397 | 110 | 0.04 | - | - | - |
0.9160 | 120 | 0.0053 | - | - | - |
0.9924 | 130 | 0.0095 | - | - | - |
1.0 | 131 | - | 0.1934 | 0.2985 | 0.2460 |
Framework Versions
- Python: 3.12.3
- Sentence Transformers: 3.3.1
- Transformers: 4.48.0.dev0
- PyTorch: 2.5.1
- Accelerate: 1.2.1
- Datasets: 3.2.0
- Tokenizers: 0.21.0
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
CachedMultipleNegativesRankingLoss
@misc{gao2021scaling,
title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
year={2021},
eprint={2101.06983},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
- Downloads last month
- 5
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for wwydmanski/modernbert-bio-v0.1
Base model
answerdotai/ModernBERT-baseEvaluation results
- Cosine Accuracy@1 on NanoNQself-reported0.100
- Cosine Accuracy@3 on NanoNQself-reported0.180
- Cosine Accuracy@5 on NanoNQself-reported0.240
- Cosine Accuracy@10 on NanoNQself-reported0.340
- Cosine Precision@1 on NanoNQself-reported0.100
- Cosine Precision@3 on NanoNQself-reported0.060
- Cosine Precision@5 on NanoNQself-reported0.048
- Cosine Precision@10 on NanoNQself-reported0.034
- Cosine Recall@1 on NanoNQself-reported0.100
- Cosine Recall@3 on NanoNQself-reported0.150