Created multiberts-seed_2-step_500k

README.md

@@ -0,0 +1,88 @@
+---
+language: en
+tags:
+-   multiberts
+-   multiberts-seed_2
+-   multiberts-seed_2-step_500k
+license: apache-2.0
+---
+
+# MultiBERTs, Intermediate Checkpoint - Seed 2, Step 500k
+
+MultiBERTs is a collection of checkpoints and a statistical library to support
+robust research on BERT. We provide 25 BERT-base models trained with
+similar hyper-parameters as
+[the original BERT model](https://github.com/google-research/bert) but
+with different random seeds, which causes variations in the initial weights and order of
+training instances. The aim is to distinguish findings that apply to a specific
+artifact (i.e., a particular instance of the model) from those that apply to the
+more general procedure.
+
+We also provide 140 intermediate checkpoints captured
+during the course of pre-training (we saved 28 checkpoints for the first 5 runs).
+
+The models were originally released through
+[http://goo.gle/multiberts](http://goo.gle/multiberts). We describe them in our
+paper
+[The MultiBERTs: BERT Reproductions for Robustness Analysis](https://arxiv.org/abs/2106.16163).
+
+This is model #2, captured at step 500k (max: 2000k, i.e., 2M steps).
+
+## Model Description
+
+This model was captured during a reproduction of
+[BERT-base uncased](https://github.com/google-research/bert), for English: it
+is a Transformers model pretrained on a large corpus of English data, using the
+Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP)
+objectives.
+
+The intended uses, limitations, training data and training procedure for the fully trained model are similar
+to [BERT-base uncased](https://github.com/google-research/bert). Two major
+differences with the original model:
+
+*   We pre-trained the MultiBERTs models for 2 million steps using sequence
+    length 512 (instead of 1 million steps using sequence length 128 then 512).
+*   We used an alternative version of Wikipedia and Books Corpus, initially
+    collected for [Turc et al., 2019](https://arxiv.org/abs/1908.08962).
+
+This is a best-effort reproduction, and so it is probable that differences with
+the original model have gone unnoticed. The performance of MultiBERTs on GLUE after full training is oftentimes comparable to that of original
+BERT, but we found significant differences on the dev set of SQuAD (MultiBERTs outperforms original BERT).
+See our [technical report](https://arxiv.org/abs/2106.16163) for more details.
+
+### How to use
+
+Using code from
+[BERT-base uncased](https://huggingface.co/bert-base-uncased), here is an example based on
+Tensorflow:
+
+```
+from transformers import BertTokenizer, TFBertModel
+tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_2-step_500k')
+model = TFBertModel.from_pretrained("google/multiberts-seed_2-step_500k")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+PyTorch version:
+
+```
+from transformers import BertTokenizer, BertModel
+tokenizer = BertTokenizer.from_pretrained('google/multiberts-seed_2-step_500k')
+model = BertModel.from_pretrained("google/multiberts-seed_2-step_500k")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+## Citation info
+
+```bibtex
+@article{sellam2021multiberts,
+  title={The MultiBERTs: BERT Reproductions for Robustness Analysis},
+  author={Thibault Sellam and Steve Yadlowsky and Jason Wei and Naomi Saphra and Alexander D'Amour and Tal Linzen and Jasmijn Bastings and Iulia Turc and Jacob Eisenstein and Dipanjan Das and Ian Tenney and Ellie Pavlick},
+  journal={arXiv preprint arXiv:2106.16163},
+  year={2021}
+}
+```

config.json

@@ -0,0 +1,25 @@
+{
+  "_name_or_path": "tmp-model",
+  "architectures": [
+    "BertForPreTraining"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.12.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d37ea66876ab3a9b1441ec3beca689cc6a0ee29bbe39945dbfc71a7b7dfcec01
+size 440509027

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba5abc0b3a2238553491c5a4df5617ecf08225304aee5f8cb61dc5f4ed54ea4a
+size 536063536

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "do_basic_tokenize": true, "never_split": null, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "tokenizer_class": "BertTokenizer"}