Update README.md

README.md

@@ -11,16 +11,19 @@ inference: false
 
 MosaicBERT-Base is a new BERT architecture and training recipe optimized for fast pretraining. 
 MosaicBERT trains faster and achieves higher pretraining and finetuning accuracy when benchmarked against 
-Hugging Face's [bert-base-uncased](https://huggingface.co/bert-base-uncased).
+Hugging Face's [bert-base-uncased](https://huggingface.co/bert-base-uncased). It incorporates efficiency insights 
+from the past half a decade of transformers research, from RoBERTa to T5 and GPT.
+
 
 __This model was trained with [ALiBi](https://arxiv.org/abs/2108.12409) on a sequence length of 1024 tokens.__
 
 ALiBi allows a model trained with a sequence length n to easily extrapolate to sequence lengths >2n during finetuning. For more details, see [Train Short, Test Long: Attention with Linear
 Biases Enables Input Length Extrapolation (Press et al. 2022)](https://arxiv.org/abs/2108.12409)
 
-It is part of the family of MosaicBERT-Base models:
+It is part of the **family of MosaicBERT-Base models** trained using ALiBi on different sequence lengths:
 
 * [mosaic-bert-base](https://huggingface.co/mosaicml/mosaic-bert-base) (trained on a sequence length of 128 tokens)
+* [mosaic-bert-base-seqlen-256](https://huggingface.co/mosaicml/mosaic-bert-base-seqlen-256)
 * [mosaic-bert-base-seqlen-512](https://huggingface.co/mosaicml/mosaic-bert-base-seqlen-512) 
 * mosaic-bert-base-seqlen-1024
 * [mosaic-bert-base-seqlen-2048](https://huggingface.co/mosaicml/mosaic-bert-base-seqlen-2048)
@@ -40,7 +43,7 @@ April 2023
 
 ```python
 from transformers import AutoModelForMaskedLM
-mlm = AutoModelForMaskedLM.from_pretrained('mosaicml/mosaic-bert-base', trust_remote_code=True)
+mlm = AutoModelForMaskedLM.from_pretrained('mosaicml/mosaic-bert-base-seqlen-1024', trust_remote_code=True)
 ```
 
 The tokenizer for this model is simply the Hugging Face `bert-base-uncased` tokenizer.
@@ -56,7 +59,7 @@ To use this model directly for masked language modeling, use `pipeline`:
 from transformers import AutoModelForMaskedLM, BertTokenizer, pipeline
 
 tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
-mlm = AutoModelForMaskedLM.from_pretrained('mosaicml/mosaic-bert-base', trust_remote_code=True)
+mlm = AutoModelForMaskedLM.from_pretrained('mosaicml/mosaic-bert-base-seqlen-1024', trust_remote_code=True)
 
 classifier = pipeline('fill-mask', model=mlm, tokenizer=tokenizer)
 
@@ -73,7 +76,7 @@ This model requires that `trust_remote_code=True` be passed to the `from_pretrai
 
 ```python
 mlm = AutoModelForMaskedLM.from_pretrained(
-   'mosaicml/mosaic-bert-base',
+   'mosaicml/mosaic-bert-base-seqlen-1024',
    trust_remote_code=True,
    revision='24512df',
 )