---
base_model: BAAI/bge-base-en-v1.5
datasets: []
language: []
library_name: sentence-transformers
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:26
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: The Supplier shall deliver the Batteries to the Manufacturer within
5 days of receipt of each
sentences:
- according to the MOU?
- What is the Delivery Schedule for the Batteries?
- single order?
- source_sentence: The Employee agrees to abide by the Employer’s rules, regulations,
guidelines, policies, and
sentences:
- When does this Agreement terminate?
- What rules and policies must the Employee abide by?
- Which law governs this Agreement, and where would disputes be resolved?
- source_sentence: 'Answer: Deepak Babbar agrees to pay Rs 5,10,000 as a full and
final settlement to Ayushi'
sentences:
- What are the Payment Terms for the Batteries?
- What financial settlement does Deepak Babbar agree to in the MOU?
- order?
- source_sentence: The Supplier agrees to supply 60,000 Batteries over the course
of one year, as specified in
sentences:
- When does the Employee commence employment with the Employer?
- When does the Company employ the Employee?
- How many Batteries are Supplier obligated to supply under this Agreement?
- source_sentence: The term of this Agreement shall continue until terminated by either
party in accordance with
sentences:
- What is the pricing per Battery under this Agreement?
- What events constitute Force Majeure under this Agreement?
- What is the term of the Agreement?
model-index:
- name: SentenceTransformer based on BAAI/bge-base-en-v1.5
results:
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 768
type: dim_768
metrics:
- type: cosine_accuracy@1
value: 0.3333333333333333
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.3333333333333333
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.3333333333333333
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.6666666666666666
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.3333333333333333
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.1111111111111111
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.06666666666666667
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.06666666666666667
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.3333333333333333
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.3333333333333333
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.3333333333333333
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.6666666666666666
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.4336766652213271
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.3703703703703704
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.3862433862433863
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 512
type: dim_512
metrics:
- type: cosine_accuracy@1
value: 0.3333333333333333
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.3333333333333333
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.3333333333333333
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.6666666666666666
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.3333333333333333
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.1111111111111111
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.06666666666666667
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.06666666666666667
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.3333333333333333
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.3333333333333333
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.3333333333333333
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.6666666666666666
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.4336766652213271
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.3703703703703704
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.38703703703703707
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 256
type: dim_256
metrics:
- type: cosine_accuracy@1
value: 0.3333333333333333
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.3333333333333333
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.3333333333333333
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.6666666666666666
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.3333333333333333
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.1111111111111111
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.06666666666666667
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.06666666666666667
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.3333333333333333
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.3333333333333333
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.3333333333333333
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.6666666666666666
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.4336766652213271
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.3703703703703704
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.38791423001949316
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 128
type: dim_128
metrics:
- type: cosine_accuracy@1
value: 0.3333333333333333
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.3333333333333333
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.3333333333333333
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 1.0
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.3333333333333333
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.1111111111111111
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.06666666666666667
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.10000000000000002
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.3333333333333333
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.3333333333333333
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.3333333333333333
name: Cosine Recall@5
- type: cosine_recall@10
value: 1.0
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.5524123942573345
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.425925925925926
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.425925925925926
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 64
type: dim_64
metrics:
- type: cosine_accuracy@1
value: 0.3333333333333333
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.6666666666666666
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.6666666666666666
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.6666666666666666
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.3333333333333333
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.2222222222222222
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.13333333333333333
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.06666666666666667
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.3333333333333333
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.6666666666666666
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.6666666666666666
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.6666666666666666
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.5
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.4444444444444444
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.47008547008547
name: Cosine Map@100
---
# SentenceTransformer based on BAAI/bge-base-en-v1.5
This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
## Model Details
### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 768 tokens
- **Similarity Function:** Cosine Similarity
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
### Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("vineet10/fm")
# Run inference
sentences = [
'The term of this Agreement shall continue until terminated by either party in accordance with',
'What is the term of the Agreement?',
'What events constitute Force Majeure under this Agreement?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```
## Evaluation
### Metrics
#### Information Retrieval
* Dataset: `dim_768`
* Evaluated with [InformationRetrievalEvaluator
](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| cosine_accuracy@1 | 0.3333 |
| cosine_accuracy@3 | 0.3333 |
| cosine_accuracy@5 | 0.3333 |
| cosine_accuracy@10 | 0.6667 |
| cosine_precision@1 | 0.3333 |
| cosine_precision@3 | 0.1111 |
| cosine_precision@5 | 0.0667 |
| cosine_precision@10 | 0.0667 |
| cosine_recall@1 | 0.3333 |
| cosine_recall@3 | 0.3333 |
| cosine_recall@5 | 0.3333 |
| cosine_recall@10 | 0.6667 |
| cosine_ndcg@10 | 0.4337 |
| cosine_mrr@10 | 0.3704 |
| **cosine_map@100** | **0.3862** |
#### Information Retrieval
* Dataset: `dim_512`
* Evaluated with [InformationRetrievalEvaluator
](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
| Metric | Value |
|:--------------------|:----------|
| cosine_accuracy@1 | 0.3333 |
| cosine_accuracy@3 | 0.3333 |
| cosine_accuracy@5 | 0.3333 |
| cosine_accuracy@10 | 0.6667 |
| cosine_precision@1 | 0.3333 |
| cosine_precision@3 | 0.1111 |
| cosine_precision@5 | 0.0667 |
| cosine_precision@10 | 0.0667 |
| cosine_recall@1 | 0.3333 |
| cosine_recall@3 | 0.3333 |
| cosine_recall@5 | 0.3333 |
| cosine_recall@10 | 0.6667 |
| cosine_ndcg@10 | 0.4337 |
| cosine_mrr@10 | 0.3704 |
| **cosine_map@100** | **0.387** |
#### Information Retrieval
* Dataset: `dim_256`
* Evaluated with [InformationRetrievalEvaluator
](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| cosine_accuracy@1 | 0.3333 |
| cosine_accuracy@3 | 0.3333 |
| cosine_accuracy@5 | 0.3333 |
| cosine_accuracy@10 | 0.6667 |
| cosine_precision@1 | 0.3333 |
| cosine_precision@3 | 0.1111 |
| cosine_precision@5 | 0.0667 |
| cosine_precision@10 | 0.0667 |
| cosine_recall@1 | 0.3333 |
| cosine_recall@3 | 0.3333 |
| cosine_recall@5 | 0.3333 |
| cosine_recall@10 | 0.6667 |
| cosine_ndcg@10 | 0.4337 |
| cosine_mrr@10 | 0.3704 |
| **cosine_map@100** | **0.3879** |
#### Information Retrieval
* Dataset: `dim_128`
* Evaluated with [InformationRetrievalEvaluator
](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| cosine_accuracy@1 | 0.3333 |
| cosine_accuracy@3 | 0.3333 |
| cosine_accuracy@5 | 0.3333 |
| cosine_accuracy@10 | 1.0 |
| cosine_precision@1 | 0.3333 |
| cosine_precision@3 | 0.1111 |
| cosine_precision@5 | 0.0667 |
| cosine_precision@10 | 0.1 |
| cosine_recall@1 | 0.3333 |
| cosine_recall@3 | 0.3333 |
| cosine_recall@5 | 0.3333 |
| cosine_recall@10 | 1.0 |
| cosine_ndcg@10 | 0.5524 |
| cosine_mrr@10 | 0.4259 |
| **cosine_map@100** | **0.4259** |
#### Information Retrieval
* Dataset: `dim_64`
* Evaluated with [InformationRetrievalEvaluator
](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| cosine_accuracy@1 | 0.3333 |
| cosine_accuracy@3 | 0.6667 |
| cosine_accuracy@5 | 0.6667 |
| cosine_accuracy@10 | 0.6667 |
| cosine_precision@1 | 0.3333 |
| cosine_precision@3 | 0.2222 |
| cosine_precision@5 | 0.1333 |
| cosine_precision@10 | 0.0667 |
| cosine_recall@1 | 0.3333 |
| cosine_recall@3 | 0.6667 |
| cosine_recall@5 | 0.6667 |
| cosine_recall@10 | 0.6667 |
| cosine_ndcg@10 | 0.5 |
| cosine_mrr@10 | 0.4444 |
| **cosine_map@100** | **0.4701** |
## Training Details
### Training Dataset
#### Unnamed Dataset
* Size: 26 training samples
* Columns: context
and question
* Approximate statistics based on the first 1000 samples:
| | context | question |
|:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|
| type | string | string |
| details |
Answer: Deepak Babbar makes the final payment of Rs 2,60,000 at the time of quashing FIR
| MOU?
|
| This Agreement is governed by the laws of Indiana, and any disputes arising out of or in
| Which law governs this Agreement, and where would disputes be resolved?
|
| Answer: After the first motion, both parties must file petitions for quashing FIRs and
| according to the MOU?
|
* Loss: [MultipleNegativesRankingLoss
](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
```json
{
"scale": 20.0,
"similarity_fct": "cos_sim"
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `eval_strategy`: steps
- `per_device_train_batch_size`: 16
- `per_device_eval_batch_size`: 16
- `num_train_epochs`: 5
- `warmup_ratio`: 0.1
- `fp16`: True
- `batch_sampler`: no_duplicates
#### All Hyperparameters