Update README.md

README.md

@@ -41,11 +41,27 @@ It achieves the following results on the evaluation set:
 
 ## Model description
 
-More information needed
+DistilBERT is a transformers model, smaller and faster than BERT, which was pretrained on the same corpus in a
+self-supervised fashion, using the BERT base model as a teacher. This means it was pretrained on the raw texts only,
+with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic
+process to generate inputs and labels from those texts using the BERT base model. More precisely, it was pretrained
+with three objectives:
+
+- Distillation loss: the model was trained to return the same probabilities as the BERT base model.
+- Masked language modeling (MLM): this is part of the original training loss of the BERT base model. When taking a
+ sentence, the model randomly masks 15% of the words in the input then run the entire masked sentence through the
+ model and has to predict the masked words. This is different from traditional recurrent neural networks (RNNs) that
+ usually see the words one after the other, or from autoregressive models like GPT which internally mask the future
+ tokens. It allows the model to learn a bidirectional representation of the sentence.
+- Cosine embedding loss: the model was also trained to generate hidden states as close as possible as the BERT base
+ model.
+
+This way, the model learns the same inner representation of the English language than its teacher model, while being
+faster for inference or downstream tasks.
 
 ## Intended uses & limitations
-
-More information needed
+[Emotion](https://huggingface.co/datasets/dair-ai/emotion) is a dataset of English Twitter messages with six basic emotions: anger, fear, joy, love, sadness, and surprise. This dataset was developed for the paper entitled "CARER: Contextualized Affect Representations for Emotion Recognition" (Saravia et al.) through noisy labels, annotated via distant
+supervision as in the paper"Twitter sentiment classification using distant supervision" (Go et al).
 
 ## Training and evaluation data