FareedKhan/flax-sentence-embeddings_all_datasets_v4_MiniLM-L6_FareedKhan_prime_synthetic_data_2k_10_64

SentenceTransformer based on flax-sentence-embeddings/all_datasets_v4_MiniLM-L6

This is a sentence-transformers model finetuned from flax-sentence-embeddings/all_datasets_v4_MiniLM-L6 on the json dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: flax-sentence-embeddings/all_datasets_v4_MiniLM-L6
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("FareedKhan/flax-sentence-embeddings_all_datasets_v4_MiniLM-L6_FareedKhan_prime_synthetic_data_2k_10_64")
# Run inference
sentences = [
    '\n\nDiarrhea, a condition characterized by the passage of loose, watery, and often more than five times a day, is a common ailment affecting individuals of all ages. It is typically acute when it lasts for a few days to a week or recurrent when it persists for more than four weeks. While acute diarrhea often resolves on its own and is usually not a cause for concern, recurrent or chronic forms require medical attention due to the risk of dehydration and nutrient deficiencies. \n\n### Causes\n\nDiarrhea can be caused by various factors, including:\n\n1. **Viral',
    'Could you assist in identifying a condition linked to congenital secretory diarrhea, similar to intractable diarrhea of infancy, given my symptoms of persistent, salty watery diarrhea, hyponatremia, abnormal body pH, and reliance on parenteral nutrition due to chronic dehydration?',
    'Could you describe the specific effects or phenotypes associated with acute hydrops in patients with the subtype of keratoconus?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_384
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.3614
cosine_accuracy@3	0.3861
cosine_accuracy@5	0.4257
cosine_accuracy@10	0.4653
cosine_precision@1	0.3614
cosine_precision@3	0.1287
cosine_precision@5	0.0851
cosine_precision@10	0.0465
cosine_recall@1	0.3614
cosine_recall@3	0.3861
cosine_recall@5	0.4257
cosine_recall@10	0.4653
cosine_ndcg@10	0.407
cosine_mrr@10	0.3891
cosine_map@100	0.396

Training Details

Training Dataset

json

• Dataset: json
• Size: 1,814 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 2 tokens mean: 118.5 tokens max: 128 tokens	min: 13 tokens mean: 35.53 tokens max: 128 tokens

• Samples:

  positive	anchor
  The list you provided appears to be a collection of various substances and medications, each with its own unique properties and uses. Here's a brief overview of each: 1. Abacavir - Used in HIV treatment, it inhibits reverse transcriptase. 2. Abate - Often refers to fenpyroximate, used as an insecticide. 3. Abidaquine - An antimalarial drug used to treat and prevent malaria. 4. Abiraterone - Used in treating prostate cancer, specifically to block the production of testosterone. 5. Abiraterone alfa - Similar to abiraterone, used in prostate cancer treatment. 6. Abiraterone acetate - An active form of abiraterone. 7. Abiraterone citrate - Another form of abiraterone. 8. Acelprozil - A medication commonly used as an anti-epileptic drug. 9. Acenocoumarol - Used as a blood thinner, also known as a vitamin K antagonist. 10. Acenocoumarol citrate - Same as acenocoumarol but with citrate, functioning similarly as a	Which pharmacological agents with antioxidant properties have the potential to disrupt the PCSK9-LDLR interaction by affecting the gene or protein players in this pathway?
  Bartholin duct cyst is a gynecological condition characterized by the distension of Bartholin glands due to mucus accumulation within the ducts, typically resulting from an obstructed orifice. This issue, categorized under women's reproductive health, falls directly under the umbrella of both integumentary system diseases and female reproductive system diseases. Originating from the Bartholin glands, which play a pivotal role in lubrication and arousal of the vulva during intercourse, the blockage or obstruction leads to cyst formation, affecting the overall female reproductive health landscape.	What is the name of the gynecological condition that arises due to blocked Bartholin's glands and involves cyst formation, falling under the broader category of women's reproductive health issues?
  Neuralgia, as defined by the MONDO ontology, refers to a pain disorder characterized by pain in the distribution of a nerve or nerves. This condition could be associated with the use of Capsaicin cream, given its known capability to alleviate symptoms by causing a temporary sensation of pain that interferes with the perception of more severe pain. Peripheral neuropathy, another symptom, is often manifest in cases where nerve damage occurs, frequently affecting multiple nerves. This condition can result in symptoms similar to sciatica, which is characterized by pain that starts in the lower back, often radiating down the leg, a common route for the sciatic nerve. The document indicates that diseases related to neuralgia include pudendal neuralgia, peripheral neuropathy, disorders involving pain, cranial neuralgia, post-infectious neuralgia, and sciatica. Furthermore, the document mentions several drugs that can be used for the purpose of managing symptoms related to neuralgia, including Lidocaine, as well as a wide array of off-label uses for treatments like Phenytoin, Morphine, Amitriptyline, Imipramine, Oxycodone, Nortriptyline, Lamotrigine, Maprotiline, Desipramine, Gabapentin, Carbamazepine, Phenobarbital, Tramadol, Venlafaxine, Trimipramine, Desvenlafaxine, Primidone, and Naltrexone.	What condition could be associated with the use of Capsaicin cream, peripheral neuropathy, and symptoms similar to sciatica?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          384
      ],
      "matryoshka_weights": [
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 64
• learning_rate: 1e-05
• num_train_epochs: 10
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_384_cosine_map@100
0	0	-	0.3614
0.3448	10	2.117	-
0.6897	20	2.1255	-
1.0	29	-	0.3855
1.0345	30	1.9375	-
1.3793	40	1.7987	-
1.7241	50	1.7494	-
2.0	58	-	0.3901
2.0690	60	1.7517	-
2.4138	70	1.676	-
2.7586	80	1.608	-
3.0	87	-	0.3934
3.1034	90	1.5923	-
3.4483	100	1.5095	-
3.7931	110	1.5735	-
4.0	116	-	0.3910
4.1379	120	1.3643	-
4.4828	130	1.4395	-
4.8276	140	1.3595	-
5.0	145	-	0.3884
5.1724	150	1.3365	-
5.5172	160	1.3506	-
5.8621	170	1.3279	-
6.0	174	-	0.3957
6.2069	180	1.3075	-
6.5517	190	1.3138	-
6.8966	200	1.2749	-
7.0	203	-	0.3979
7.2414	210	1.1725	-
7.5862	220	1.2696	-
7.9310	230	1.2487	-
8.0	232	-	0.3986
8.2759	240	1.1558	-
8.6207	250	1.2447	-
8.9655	260	1.2566	-
9.0	261	-	0.3964
9.3103	270	1.2493	-
9.6552	280	1.2697	-
10.0	290	1.079	0.3960

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.10
• Sentence Transformers: 3.1.1
• Transformers: 4.45.1
• PyTorch: 2.2.1+cu121
• Accelerate: 0.34.2
• Datasets: 3.0.1
• Tokenizers: 0.20.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}