Update README.md

README.md

@@ -8,3 +8,61 @@ widget:
 - text: 0.05 ML aflibercept 40 MG/ML Injection is contraindicated with [MASK].
 - text: The most import ingrediant for Excedrin is [MASK].
 ---
+
+## Overview 
+
+This repository contains the bert_base_uncased_rxnorm_babbage model, a continually pretrained [Bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased) model with drugs, diseases, and their relationships from RxNorm using masked language modeling. 
+We hypothesize that the augmentation can boost the model's understanding of medical terminologies and contexts.
+
+It uses a corpus comprising approximately 8.8M million tokens sythesized using drug and disease relations harvested from RxNorm. A few exampes show below.
+```plaintext
+ferrous fumarate 191 MG is contraindicated with Hemochromatosis.
+24 HR metoprolol succinate 50 MG Extended Release Oral Capsule [Kapspargo] contains the ingredient Metoprolol.
+Genvoya has the established pharmacologic class Cytochrome P450 3A Inhibitor.
+cefprozil 250 MG Oral Tablet may be used to treat Haemophilus Infections.
+mecobalamin 1 MG Sublingual Tablet contains the ingredient Vitamin B 12.
+```
+
+The dataset is hosted at [this commit](https://github.com/Su-informatics-lab/drug_disease_graph/blob/3a598cb9d55ffbb52d2f16e61eafff4dfefaf5b1/rxnorm.txt). 
+Note, this is the babbage version of the corpus using *all* drug and disease relations. 
+Don't confuse it with the ada version, where only a fraction of the relationships are used (see [the repo](https://github.com/Su-informatics-lab/drug_disease_graph/tree/main) for more information).
+
+## Training
+
+15% of the data was masked for prediction. 
+The model processes this data for *20* epochs.
+Training happens on 4 A40(48G) using python3.8 (tried to match up dependencies specified at [requirements.txt](https://github.com/Su-informatics-lab/rxnorm_gatortron/blob/1f15ad349056e22089118519becf1392df084701/requirements.txt)). 
+It has a batch size of 16 and a learning rate of 5e-5.
+See more configuration at [GitHub](https://github.com/Su-informatics-lab/rxnorm_gatortron/blob/main/runs_mlm_bert_base_uncased_rxnorm_babbage.sh) and training curves at [WandB](https://wandb.ai/hainingwang/continual_pretraining_gatortron/runs/1abzfvb9).
+
+
+## Usage 
+You can use this model for masked language modeling tasks to predict missing words in a given text.
+Below are the instructions and examples to get you started.
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+import torch
+
+# load the tokenizer and model
+tokenizer = AutoTokenizer.from_pretrained("Su-informatics-lab/bert_base_uncased_rxnorm_babbage")
+model = AutoModelForMaskedLM.from_pretrained("Su-informatics-lab/bert_base_uncased_rxnorm_babbage")
+
+# prepare the input
+text = "0.05 ML aflibercept 40 MG/ML Injection is contraindicated with [MASK]."
+inputs = tokenizer(text, return_tensors="pt")
+
+# get model predictions
+with torch.no_grad():
+    outputs = model(**inputs)
+
+# decode the predictions
+predictions = outputs.logits
+masked_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]
+predicted_token_id = predictions[0, masked_index].argmax(axis=-1)
+predicted_token = tokenizer.decode(predicted_token_id)
+```
+
+## License
+Apache 2.0.
+