Update README.md

README.md

@@ -1,4 +1,93 @@
----
-license: cc-by-4.0
----
-Please direct all questions to [email protected]
+---
+license: cc-by-4.0
+---
+# Keyphrase Boundary Infilling with Replacement (KBIR)
+The KBIR model as described in Learning Rich Representations of Keyphrases from Text (https://arxiv.org/pdf/2112.08547.pdf) builds on top of the RoBERTa architecture by adding an Infilling head and a Replacement Classification head that is used during pre-training. However, these heads are not used during the downstream evaluation of the model and we only leverage the pre-trained embeddings. Discarding the heads thereby allows us to be compatible with all AutoModel classes that RoBERTa supports.
+
+We provide examples on how to perform downstream evaluation on some of the tasks reported in the paper.
+## Downstream Evaluation
+
+### Keyphrase Extraction
+```
+from transformers import AutoTokenizer, AutoModelForTokenClassification
+
+tokenizer = AutoTokenizer.from_pretrained("bloomberg/KBIR")
+model = AutoModelForTokenClassification.from_pretrained("bloomberg/KBIR")
+
+from datasets import load_dataset
+
+dataset = load_dataset("midas/semeval2017_ke_tagged")
+```
+
+Reported Results:
+
+| Model                 | Inspec | SE10  | SE17  |
+|-----------------------|--------|-------|-------|
+| RoBERTa+BiLSTM-CRF    | 59.5   | 27.8  | 50.8  |
+| RoBERTa+TG-CRF        | 60.4   | 29.7  | 52.1  |
+| SciBERT+Hypernet-CRF  | 62.1   | 36.7  | 54.4  |
+| RoBERTa+Hypernet-CRF  | 62.3   | 34.8  | 53.3  |
+| RoBERTa-extended-CRF* | 62.09  | 40.61 | 52.32 |
+| KBI-CRF*              | 62.61  | 40.81 | 59.7  |
+| KBIR-CRF*             | 62.72  | 40.15 | 62.56 |
+
+### Named Entity Recognition
+```
+from transformers import AutoTokenizer, AutoModelForTokenClassification
+
+tokenizer = AutoTokenizer.from_pretrained("bloomberg/KBIR")
+model = AutoModelForTokenClassification.from_pretrained("bloomberg/KBIR")
+
+from datasets import load_dataset
+
+dataset = load_dataset("conll2003")
+```
+
+Reported Results:
+
+| Model                           | F1    |
+|---------------------------------|-------|
+| LSTM-CRF (Lample et al., 2016)  | 91.0  |
+| ELMo (Peters et al., 2018)      | 92.2  |
+| BERT (Devlin et al., 2018)      | 92.8  |
+| (Akbik et al., 2019)            | 93.1  |
+| (Baevski et al., 2019)          | 93.5  |
+| LUKE (Yamada et al., 2020)      | 94.3  |
+| LUKE w/o entity attention       | 94.1  |
+| RoBERTa (Yamada et al., 2020)   | 92.4  |
+| RoBERTa-extended*               | 92.54 |
+| KBI*                            | 92.73 |
+| KBIR*                           | 92.97 |
+
+### Question Answering
+```
+from transformers import AutoTokenizer, AutoModelForQuestionAnswering
+
+tokenizer = AutoTokenizer.from_pretrained("bloomberg/KBIR")
+model = AutoModelForQuestionAnswering.from_pretrained("bloomberg/KBIR")
+
+from datasets import load_dataset
+
+dataset = load_dataset("squad")
+```
+Reported Results:
+
+| Model                  | EM    | F1    |
+|------------------------|-------|-------|
+| BERT                   | 84.2  | 91.1  |
+| XLNet                  | 89.0  | 94.5  |
+| ALBERT                 | 89.3  | 94.8  |
+| LUKE                   | 89.8  | 95.0  |
+| LUKE w/o entity attention | 89.2  | 94.7  |
+| RoBERTa                | 88.9  | 94.6  |
+| RoBERTa-extended*      | 88.88 | 94.55 |
+| KBI*                   | 88.97 | 94.7  |
+| KBIR*                  | 89.04 | 94.75 |
+
+## Any other classification task
+As mentioned above since KBIR is built on top of the RoBERTa architecture, it is compatible with any AutoModel setting that RoBERTa is also compatible with. 
+
+We encourage you to try fine-tuning KBIR on different datasets and report the downstream results.
+
+## Contact
+For any questions contact [email protected]