NoInstruct-small-Embedding-v0-nli

NoInstruct-small-Embedding-v0-nli is a fine-tuned NLI model that classifies the relationship between pairs of sentences into three categories: entailment, neutral, and contradiction. It enhances the capabilities of avsolatorio/NoInstruct-small-Embedding-v0 for improved performance on NLI tasks.

Intended Use

NoInstruct-small-Embedding-v0-nli is ideal for applications requiring understanding of logical relationships between sentences, including:

  • Semantic textual similarity
  • Question answering
  • Dialogue systems
  • Content moderation

Performance

NoInstruct-small-Embedding-v0-nli was trained on the sentence-transformers/all-nli dataset, achieving competitive results in sentence pair classification.

Performance on the MNLI matched validation set:

  • Accuracy: 0.7687
  • Precision: 0.77
  • Recall: 0.77
  • F1-score: 0.77

Training details

Training Details
  • Dataset:

  • Sampling:

    • 100 000 training samples and 10 000 evaluation samples.
  • Fine-tuning Process:

    • Custom Python script with adaptive precision training (bfloat16).
    • Early stopping based on evaluation loss.
  • Hyperparameters:

    • Learning Rate: 2e-5
    • Batch Size: 64
    • Optimizer: AdamW (weight decay: 0.01)
    • Training Duration: Up to 10 epochs
Reproducibility

To ensure reproducibility:

  • Fixed random seed: 42
  • Environment:
    • Python: 3.10.12
    • PyTorch: 2.5.1
    • Transformers: 4.44.2

Usage Instructions

Using Sentence Transformers

from sentence_transformers import CrossEncoder

model_name = "agentlans/NoInstruct-small-Embedding-v0-nli"
model = CrossEncoder(model_name)
scores = model.predict(
    [
        ("A man is eating pizza", "A man eats something"),
        (
            "A black race car starts up in front of a crowd of people.",
            "A man is driving down a lonely road.",
        ),
    ]
)

label_mapping = ["entailment", "neutral", "contradiction"]
labels = [label_mapping[score_max] for score_max in scores.argmax(axis=1)]
print(labels)

Using Transformers Library

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "agentlans/NoInstruct-small-Embedding-v0-nli"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

features = tokenizer(
    [
        "A man is eating pizza",
        "A black race car starts up in front of a crowd of people.",
    ],
    ["A man eats something", "A man is driving down a lonely road."],
    padding=True,
    truncation=True,
    return_tensors="pt",
)

model.eval()
with torch.no_grad():
    scores = model(**features).logits
    label_mapping = ["entailment", "neutral", "contradiction"]
    labels = [label_mapping[score_max] for score_max in scores.argmax(dim=1)]
    print(labels)

Limitations and Ethical Considerations

NoInstruct-small-Embedding-v0-nli may reflect biases present in the training data. Users should evaluate its performance in specific contexts to ensure fairness and accuracy.

Conclusion

NoInstruct-small-Embedding-v0-nli offers a robust solution for NLI tasks, enhancing avsolatorio/NoInstruct-small-Embedding-v0's capabilities with straightforward integration into existing frameworks. It aids developers in building intelligent applications that require nuanced language understanding.

Downloads last month
10
Safetensors
Model size
33.4M params
Tensor type
F32
·
Inference Examples
Inference API (serverless) does not yet support sentence-transformers models for this pipeline type.