SetFit with sentence-transformers/all-MiniLM-L6-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/all-MiniLM-L6-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: sentence-transformers/all-MiniLM-L6-v2
Classification head: a LogisticRegression instance
Maximum Sequence Length: 256 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
1	'In January, as part of its advocacy for the protection of civilians and human rights, the United Nations Joint Human Rights Office in the Democratic Republic of the Congo issued two public reports highlighting the upward trend in human rights violations and abuses committed in Ituri and North Kivu by armed groups, as well as by members of the national security and defence forces.' 'A son indépendance, en 1960, la RDC avait un PIB par habitant de 325 USD et était la deuxième économie la plus industrialisée d’Afrique, après l’Afrique du Sud.' "Les populations les plus gravement touchées sont celles qui ont été déplacées, les groupes de réfugiés et de populations rentrées chez elles, les familles d'accueil et les populations victimes de catastrophes naturelles (inondations, glissements de terrain, incendies) ainsi que les ménages dont le chef de famille est une femme."
0	'This may be driven by children’s varying levels of education and their different language skills,' 'Ce sont des travaux très pénibles qui nuisent à leur santé physique.' 'Screening and treatment of MAM were enabled for 10,184 children aged 6-59 months and 2,613 PLW.'

Label

Examples

'In January, as part of its advocacy for the protection of civilians and human rights, the United Nations Joint Human Rights Office in the Democratic Republic of the Congo issued two public reports highlighting the upward trend in human rights violations and abuses committed in Ituri and North Kivu by armed groups, as well as by members of the national security and defence forces.'
'A son indépendance, en 1960, la RDC avait un PIB par habitant de 325 USD et était la deuxième économie la plus industrialisée d’Afrique, après l’Afrique du Sud.'
"Les populations les plus gravement touchées sont celles qui ont été déplacées, les groupes de réfugiés et de populations rentrées chez elles, les familles d'accueil et les populations victimes de catastrophes naturelles (inondations, glissements de terrain, incendies) ainsi que les ménages dont le chef de famille est une femme."

'This may be driven by children’s varying levels of education and their different language skills,'
'Ce sont des travaux très pénibles qui nuisent à leur santé physique.'
'Screening and treatment of MAM were enabled for 10,184 children aged 6-59 months and 2,613 PLW.'

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("Selon ces PDIs, des parents restés ou retournés au village les auraient informées de l’amélioration de la situation sécuritaire.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	1	25.2763	95

Label	Training Sample Count
0	295
1	313

Training Hyperparameters

batch_size: (32, 32)
num_epochs: (1, 1)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 35
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0008	1	0.4533	-
0.0376	50	0.3371	-
0.0752	100	0.2585	-
0.1128	150	0.2574	-
0.1504	200	0.2535	-
0.1880	250	0.2513	-
0.2256	300	0.2573	-
0.2632	350	0.246	-
0.3008	400	0.2471	-
0.3383	450	0.247	-
0.3759	500	0.2348	-
0.4135	550	0.2165	-
0.4511	600	0.1911	-
0.4887	650	0.1402	-
0.5263	700	0.0865	-
0.5639	750	0.049	-
0.6015	800	0.0279	-
0.6391	850	0.0188	-
0.6767	900	0.0108	-
0.7143	950	0.0072	-
0.7519	1000	0.0051	-
0.7895	1050	0.0039	-
0.8271	1100	0.0032	-
0.8647	1150	0.0039	-
0.9023	1200	0.0025	-
0.9398	1250	0.0024	-
0.9774	1300	0.0023	-

Framework Versions

Python: 3.11.5
SetFit: 1.1.0
Sentence Transformers: 3.1.1
Transformers: 4.45.1
PyTorch: 2.1.0
Datasets: 2.17.1
Tokenizers: 0.20.0

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

Sfekih
/

sentence_independancy_model