Update README.md

README.md

@@ -12,6 +12,8 @@ widget:
 
 This is the set of 24 Chinese RoBERTa models pre-trained by [UER-py](https://www.aclweb.org/anthology/D19-3041.pdf).
 
+[Turc et al.](https://arxiv.org/abs/1908.08962) have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released 24 Chinese RoBERTa models. In order to facilitate users to reproduce the results, we used the publicly available corpus and provided all training details.
+
 You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Github page](https://github.com/dbiir/UER-py/), or via HuggingFace from the links below:
 
 |   |H=128|H=256|H=512|H=768|
@@ -21,7 +23,22 @@ You can download the 24 Chinese RoBERTa miniatures either from the [UER-py Githu
 | **L=6**  |[6/128]|[6/256]|[6/512]|[6/768]|
 | **L=8**  |[8/128]|[8/256]|[**8/512 (Medium)**][8_512]|[8/768]|
 | **L=10** |[10/128]|[10/256]|[10/512]|[10/768]|
-| **L=12** |[12/128]|[12/256]|[12/512]|[**12/768 (Base)**]|
+| **L=12** |[12/128]|[12/256]|[12/512]|[**12/768 (Base)**][12_768]|
+
+Here are scores on the devlopment set of six Chinese tasks:
+
+|Model|Score|douban|chnsenticorp|lcqmc|tnews(CLUE)|iflytek(CLUE)|ocnli(CLUE)|
+|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+|BERT-Tiny|0.0|0.0|83.2|0.0|0.0|0.0|70.2|
+|BERT-Mini|0.0|0.0|85.9|0.0|75.4/73.3|0.0|70.2|
+|BERT-Small|0.0|27.8|89.7|83.4|78.8|68.1|77.6|
+|BERT-Medium|0.0|38.0|89.6|86.6|80.4|69.6|80.0|
+|BERT-Base|0.0|38.0|89.6|86.6|80.4|69.6|80.0|
+
+For each task, we selected the best fine-tuning hyperparameters from the lists below:
+- epochs: 3, 5, 8
+- batch sizes: 32, 64
+- learning rates: 3e-5, 1e-4, 3e-4
 
 ## How to use
 
@@ -81,7 +98,7 @@ output = model(encoded_input)
 
 ## Training data
 
-CLUECorpus2020 and CLUECorpusSmall are used as training data.
+CLUECorpusSmall is used as training data. We found that models pre-trained on CLUECorpusSmall outperform those pre-trained on CLUECorpus2020, although CLUECorpus2020 is much larger than CLUECorpusSmall.
 
 ## Training procedure
 
@@ -89,40 +106,40 @@ Models are pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tencent
 
 Stage1:
 ```
-python3 preprocess.py --corpus_path corpora/cluecorpus.txt \
+python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                       --vocab_path models/google_zh_vocab.txt \
-					  --dataset_path cluecorpus_seq128_dataset.pt \
+					  --dataset_path cluecorpussmall_seq128_dataset.pt \
 					  --processes_num 32 --seq_length 128 \
 					  --dynamic_masking --target mlm
 ```
 ```
-python3 pretrain.py --dataset_path cluecorpus_seq128_dataset.pt \
+python3 pretrain.py --dataset_path cluecorpussmall_seq128_dataset.pt \
                     --vocab_path models/google_zh_vocab.txt \
-					--config_path models/bert_small_config.json \
-					--output_model_path models/cluecorpus_roberta_small_seq128_model.bin \
+					--config_path models/bert_xxx_config.json \
+					--output_model_path models/cluecorpussmall_roberta_xxxx_seq128_model.bin \
 					--world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
 					--total_steps 1000000 --save_checkpoint_steps 100000 --report_steps 50000 \
 					--learning_rate 1e-4 --batch_size 64 \
-					--tie_weights --encoder bert --target mlm
+					--tie_weights --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm
 ```
 Stage2:
 ```
-python3 preprocess.py --corpus_path corpora/cluecorpus.txt \
+python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
                       --vocab_path models/google_zh_vocab.txt \
-					  --dataset_path cluecorpus_seq512_dataset.pt \
+					  --dataset_path cluecorpussmall_seq512_dataset.pt \
 					  --processes_num 32 --seq_length 512 \
 					  --dynamic_masking --target mlm
 ```
 ```
-python3 pretrain.py --dataset_path cluecorpus_seq512_dataset.pt \
-                    --pretrained_model_path models/cluecorpus_roberta_small_seq128_model.bin-1000000 \
+python3 pretrain.py --dataset_path cluecorpussmall_seq512_dataset.pt \
+                    --pretrained_model_path models/cluecorpussmall_roberta_xxx_seq128_model.bin-1000000 \
 					--vocab_path models/google_zh_vocab.txt \
-					--config_path models/bert_small_config.json \
-					--output_model_path models/cluecorpus_roberta_small_seq512_model.bin \
+					--config_path models/bert_xxx_config.json \
+					--output_model_path models/cluecorpussmall_roberta_xxx_seq512_model.bin \
 					--world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
 					--total_steps 250000 --save_checkpoint_steps 50000 --report_steps 10000 \
 					--learning_rate 5e-5 --batch_size 16 \
-					--tie_weights --encoder bert --target mlm
+					--tie_weights --embedding word_pos_seg --encoder transformer --mask fully_visible --target mlm
 ```
 
 ### BibTeX entry and citation info
@@ -139,6 +156,7 @@ python3 pretrain.py --dataset_path cluecorpus_seq512_dataset.pt \
 
 [2_128]: https://huggingface.co/uer/chinese_roberta_L-2_H-128
 [4_256]: https://huggingface.co/uer/chinese_roberta_L-4_H-256
-[8_512]: https://huggingface.co/uer/chinese_roberta_L-8_H-512
 [4_512]: https://huggingface.co/uer/chinese_roberta_L-4_H-512
+[8_512]: https://huggingface.co/uer/chinese_roberta_L-8_H-512
+[12_768]: https://huggingface.co/uer/chinese_roberta_L-12_H-768