update readme

1_Pooling/config.json

@@ -5,6 +5,5 @@
   "pooling_mode_max_tokens": false,
   "pooling_mode_mean_sqrt_len_tokens": false,
   "pooling_mode_weightedmean_tokens": false,
-  "pooling_mode_lasttoken": false,
-  "include_prompt": true
+  "pooling_mode_lasttoken": false
 }

README.md

@@ -1,34 +1,54 @@
 ---
+language: fr
+license: mit
 library_name: sentence-transformers
-pipeline_tag: sentence-similarity
+pipeline_tag: feature-extraction
 tags:
-- sentence-transformers
-- feature-extraction
-- sentence-similarity
-- transformers
+    - sentence-transformers
+    - feature-extraction
+    - sentence-similarity
+    - transformers
 datasets:
-- stsb_multi_mt
+    - stsb_multi_mt
+metrics:
+    - pearsonr
+base_model: almanach/camembert-base
+model-index:
+  - name: sts-distilcamembert-base
+    results:
+      - task:
+          name: Sentence Similarity
+          type: sentence-similarity
+        dataset:
+          name: STSb French
+          type: stsb_multi_mt
+          args: fr
+        metrics:
+          - name: Pearson Correlation - stsb_multi_mt fr
+            type: pearsonr
+            value: 0.8165
 ---
 
-# h4c5/sts-distilcamembert-base
+## Description
 
-This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
+Ce modèle [sentence-transformers](https://www.SBERT.net) a été obtenu en finetunant le modèle 
+[`almanach/camembert-base`](https://huggingface.co/almanach/camembert-base) à l'aide de la librairie 
+[sentence-transformers](https://www.SBERT.net).
 
-<!--- Describe your model here -->
+Il permet d'encoder une phrase ou un pararaphe (514 tokens maximum) en un vecteur de dimension 768.
 
-## Usage (Sentence-Transformers)
+Le modèle [CamemBERT](https://arxiv.org/abs/1911.03894) sur lequel il est basé est un modèle de type RoBERTa qui est 
+à l'état de l'art pour la langue française.
 
-Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+## Utilisation via la librairie `sentence-transformers`
 
 ```
 pip install -U sentence-transformers
 ```
 
-Then you can use the model like this:
-
 ```python
 from sentence_transformers import SentenceTransformer
-sentences = ["This is an example sentence", "Each sentence is converted"]
+sentences = ["Ceci est un exemple", "deuxième exemple"]
 
 model = SentenceTransformer('h4c5/sts-distilcamembert-base')
 embeddings = model.encode(sentences)
@@ -36,50 +56,86 @@ print(embeddings)
 ```
 
 
+## Utilisation via la librairie `transformers`
 
-## Usage (HuggingFace Transformers)
-Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
+```
+pip install -U transformers
+```
 
 ```python
 from transformers import AutoTokenizer, AutoModel
 import torch
 
+tokenizer = AutoTokenizer.from_pretrained("h4c5/sts-distilcamembert-base")
+model = AutoModel.from_pretrained("h4c5/sts-distilcamembert-base")
+model.eval()
+
 
-#Mean Pooling - Take attention mask into account for correct averaging
+# Mean Pooling
 def mean_pooling(model_output, attention_mask):
-    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
-    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
-    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+    token_embeddings = model_output[
+        0
+    ]  # First element of model_output contains all token embeddings
+    input_mask_expanded = (
+        attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    )
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(
+        input_mask_expanded.sum(1), min=1e-9
+    )
+
+# Tokenization et calcul des embeddings des tokens
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")
+model_output = model(**encoded_input)
+
+# Mean pooling
+sentence_embeddings = mean_pooling(model_output, encoded_input["attention_mask"])
 
+print(sentence_embeddings)
+```
 
-# Sentences we want sentence embeddings for
-sentences = ['This is an example sentence', 'Each sentence is converted']
 
-# Load model from HuggingFace Hub
-tokenizer = AutoTokenizer.from_pretrained('h4c5/sts-distilcamembert-base')
-model = AutoModel.from_pretrained('h4c5/sts-distilcamembert-base')
+## Evaluation
 
-# Tokenize sentences
-encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+Le modèle a été évalué sur le jeu de données [STSb fr](https://huggingface.co/datasets/stsb_multi_mt) : 
 
-# Compute token embeddings
-with torch.no_grad():
-    model_output = model(**encoded_input)
+```python
+from datasets import load_dataset
+from sentence_transformers import InputExample, evaluation
 
-# Perform pooling. In this case, mean pooling.
-sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
 
-print("Sentence embeddings:")
-print(sentence_embeddings)
-```
+def dataset_to_input_examples(dataset):
+    return [
+        InputExample(
+            texts=[example["sentence1"], example["sentence2"]],
+            label=example["similarity_score"] / 5.0,
+        )
+        for example in dataset
+    ]
+
+
+sts_test_dataset = load_dataset("stsb_multi_mt", name="fr", split="test")
+sts_test_examples = dataset_to_input_examples(sts_test_dataset)
+
+sts_test_evaluator = evaluation.EmbeddingSimilarityEvaluator.from_input_examples(
+    sts_test_examples, name="sts-test"
+)
 
+sts_test_evaluator(model, ".")
+```
 
+### Résultats
 
-## Evaluation Results
+Ci-dessous, les résultats de l'évaluation du modèle sur le jeu données [`stsb_multi_mt`](https://huggingface.co/datasets/stsb_multi_mt)
+(données `fr`, split `test`)
 
-<!--- Describe how your model was evaluated -->
+| Model                                                                                                                                          | Pearson Correlation | Paramètres |
+| :--------------------------------------------------------------------------------------------------------------------------------------------- | :-----------------: | ---------: |
+| [`h4c5/sts-camembert-base`](https://huggingface.co/h4c5/sts-camembert-base)                                                                    |      **0.837**      |       110M |
+| [`Lajavaness/sentence-camembert-base`](https://huggingface.co/Lajavaness/sentence-camembert-base)                                              |        0.835        |       110M |
+| [`inokufu/flaubert-base-uncased-xnli-sts`](https://huggingface.co/inokufu/flaubert-base-uncased-xnli-sts)                                      |        0.828        |       137M |
+| [`h4c5/sts-distilcamembert-base`](https://huggingface.co/h4c5/sts-distilcamembert-base)                                                        |        0.817        |        64M |
+| [`sentence-transformers/distiluse-base-multilingual-cased-v2`](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased) |        0.786        |       135M |
 
-For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=h4c5/sts-distilcamembert-base)
 
 
 ## Training
@@ -96,11 +152,11 @@ The model was trained with the parameters:
 
 `sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss` 
 
-Parameters of the fit()-Method:
+Parameters of the `fit()` method:
 ```
 {
     "epochs": 10,
-    "evaluation_steps": 0,
+    "evaluation_steps": 1000,
     "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
     "max_grad_norm": 1,
     "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
@@ -116,6 +172,7 @@ Parameters of the fit()-Method:
 
 
 ## Full Model Architecture
+
 ```
 SentenceTransformer(
   (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: CamembertModel 
@@ -123,6 +180,45 @@ SentenceTransformer(
 )
 ```
 
-## Citing & Authors
-
-<!--- Describe where people can find more information -->
+## Citing
+
+    @inproceedings{reimers-2019-sentence-bert,
+        title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+        author = "Reimers, Nils and Gurevych, Iryna",
+        booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+        month = "11",
+        year = "2019",
+        publisher = "Association for Computational Linguistics",
+        journal={"https://arxiv.org/abs/1908.10084"},
+    }
+
+    @inproceedings{sanh2019distilbert,
+        title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
+        author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},
+        booktitle={NeurIPS EMC^2 Workshop},
+        journal={https://arxiv.org/abs/1910.01108},
+        year={2019}
+    }
+
+    @inproceedings{martin2020camembert,
+        title={CamemBERT: a Tasty French Language Model},
+        author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
+        booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
+        journal={https://arxiv.org/abs/1911.03894},
+        year={2020}
+    }
+
+    @inproceedings{delestre:hal-03674695,
+        TITLE = {{DistilCamemBERT : une distillation du mod{\`e}le fran{\c c}ais CamemBERT}},
+        AUTHOR = {Delestre, Cyrile and Amar, Abibatou},
+        URL = {https://hal.archives-ouvertes.fr/hal-03674695},
+        BOOKTITLE = {{CAp (Conf{\'e}rence sur l'Apprentissage automatique)}},
+        ADDRESS = {Vannes, France},
+        YEAR = {2022},
+        MONTH = Jul,
+        KEYWORDS = {NLP ; Transformers ; CamemBERT ; Distillation},
+        PDF = {https://hal.archives-ouvertes.fr/hal-03674695/file/cap2022.pdf},
+        HAL_ID = {hal-03674695},
+        HAL_VERSION = {v1},
+        journal={https://arxiv.org/abs/2205.11111},
+    }

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "cmarkea/distilcamembert-base",
+  "_name_or_path": "h4c5/sts-distilcamembert-base",
   "architectures": [
     "CamembertModel"
   ],
@@ -25,4 +25,4 @@
   "type_vocab_size": 1,
   "use_cache": true,
   "vocab_size": 32005
-}
+}