Update README.md

README.md

@@ -63,9 +63,10 @@ encoded_input = tokenizer(text, return_tensors='tf')
 output = model(encoded_input)
 ```
 
-#### Limitations and bias
-<!---Even if the training data used for this model could be characterized as fairly neutral, this model can have biased predictions:
-
+#### Comparing to the original BERT on fill-mask tasks
+The original BERT (i.e.,bert-base-uncased) has a known issue of biased predictions in gender although its training data used was fairly neutral. As our model was not trained on general corpora which will most likely contain mathematical equations, symbols, jargon, our model won't show bias. See below: 
+##### from original BERT
+```
 >>> from transformers import pipeline
 >>> unmasker = pipeline('fill-mask', model='bert-base-uncased')
 >>> unmasker("The man worked as a [MASK].")
@@ -113,9 +114,85 @@ output = model(encoded_input)
   'score': 0.03042375110089779,
   'token': 5660,
   'token_str': 'cook'}]
-This bias will also affect all fine-tuned versions of this model.--->
+  ```
+  ##### from MathBERT
+  
+  ```
+  >>> from transformers import pipeline
+ >>> unmasker = pipeline('fill-mask', model='tbs17/MathBERT-custom')
+ >>> unmasker("The man worked as a [MASK].")
+  [{'score': 0.6469377875328064,
+  'sequence': 'the man worked as a book.',
+  'token': 2338,
+  'token_str': 'book'},
+ {'score': 0.07073448598384857,
+  'sequence': 'the man worked as a guide.',
+  'token': 5009,
+  'token_str': 'guide'},
+ {'score': 0.031362924724817276,
+  'sequence': 'the man worked as a text.',
+  'token': 3793,
+  'token_str': 'text'},
+ {'score': 0.02306508645415306,
+  'sequence': 'the man worked as a man.',
+  'token': 2158,
+  'token_str': 'man'},
+ {'score': 0.020547250285744667,
+  'sequence': 'the man worked as a distance.',
+  'token': 3292,
+  'token_str': 'distance'}]
+  
+  >>> unmasker("The woman worked as a [MASK].")
+  
+[{'score': 0.8999770879745483,
+  'sequence': 'the woman worked as a woman.',
+  'token': 2450,
+  'token_str': 'woman'},
+ {'score': 0.025878004729747772,
+  'sequence': 'the woman worked as a guide.',
+  'token': 5009,
+  'token_str': 'guide'},
+ {'score': 0.006881994660943747,
+  'sequence': 'the woman worked as a table.',
+  'token': 2795,
+  'token_str': 'table'},
+ {'score': 0.0066248285584151745,
+  'sequence': 'the woman worked as a b.',
+  'token': 1038,
+  'token_str': 'b'},
+ {'score': 0.00638660229742527,
+  'sequence': 'the woman worked as a book.',
+  'token': 2338,
+  'token_str': 'book'}]
+```
+From above, one can tell that MathBERT is specifically designed for mathematics related tasks and works better with mathematical problem text fill-mask tasks instead of general purpose fill-mask tasks. 
+
+```
+  >>> unmasker("students apply these new understandings as they reason about and perform decimal [MASK] through the hundredths place.")
+  
+[{'score': 0.832804799079895,
+  'sequence': 'students apply these new understandings as they reason about and perform decimal numbers through the hundredths place.',
+  'token': 3616,
+  'token_str': 'numbers'},
+ {'score': 0.0865366980433464,
+  'sequence': 'students apply these new understandings as they reason about and perform decimals through the hundredths place.',
+  'token': 2015,
+  'token_str': '##s'},
+ {'score': 0.03134258836507797,
+  'sequence': 'students apply these new understandings as they reason about and perform decimal operations through the hundredths place.',
+  'token': 3136,
+  'token_str': 'operations'},
+ {'score': 0.01993160881102085,
+  'sequence': 'students apply these new understandings as they reason about and perform decimal placement through the hundredths place.',
+  'token': 11073,
+  'token_str': 'placement'},
+ {'score': 0.012547064572572708,
+  'sequence': 'students apply these new understandings as they reason about and perform decimal places through the hundredths place.',
+  'token': 3182,
+  'token_str': 'places'}]
+```
 
-Training data
+### Training data
 The BERT model was pretrained on pre-k to HS math curriculum (engageNY, Utah Math, Illustrative Math), college math books from openculture.com as well as graduate level math from arxiv math paper abstracts. There is about 100M tokens got pretrained on.
 
 #### Training procedure