version 2: trained on CC100 and Wikipedia

README.md

@@ -11,6 +11,7 @@ tags:
 license: mit
 datasets:
 - cc100
+- wikipedia
 ---
 
 # japanese-gpt2-medium
@@ -36,7 +37,7 @@ model = AutoModelForCausalLM.from_pretrained("rinna/japanese-gpt2-medium")
 A 24-layer, 1024-hidden-size transformer-based language model.
 
 # Training
-The model was trained on [Japanese CC-100](http://data.statmt.org/cc-100/ja.txt.xz) to optimize a traditional language modelling objective on 8\\*V100 GPUs for around 30 days. It reaches around 18 perplexity on a chosen validation set from the same data.
+The model was trained on [Japanese CC-100](http://data.statmt.org/cc-100/ja.txt.xz) and [Japanese Wikipedia](https://dumps.wikimedia.org/other/cirrussearch) to optimize a traditional language modelling objective on 8\\*V100 GPUs for around 30 days. It reaches around 18 perplexity on a chosen validation set from the same data.
 
 # Tokenization
 The model uses a [sentencepiece](https://github.com/google/sentencepiece)-based tokenizer, the vocabulary was trained on the Japanese Wikipedia using the official sentencepiece training script.

config.json

@@ -7,8 +7,10 @@
   "bos_token_id": 1,
   "embd_pdrop": 0.1,
   "eos_token_id": 2,
+  "gradient_checkpointing": false,
   "initializer_range": 0.02,
   "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
   "n_ctx": 1024,
   "n_embd": 1024,
   "n_head": 16,
@@ -16,6 +18,7 @@
   "n_layer": 24,
   "n_positions": 1024,
   "resid_pdrop": 0.1,
+  "scale_attn_weights": true,
   "summary_activation": null,
   "summary_first_dropout": 0.1,
   "summary_proj_to_labels": true,
@@ -27,5 +30,7 @@
       "max_length": 50
     }
   },
+  "transformers_version": "4.8.2",
+  "use_cache": true,
   "vocab_size": 32000
 }

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f558067e8d3801c5535b679e3fc53d87d8aa8ded989eaf44ed0cae491f259a60
-size 1369804727
+oid sha256:e260ffe881d681a84d78e80c4e5c0b385c32158b7528475ff4c3f462e67e779b
+size 1369804067

tf_model.h5

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1aaae55e81e5704d4d4d5fe38bced6ac1d9f5434e95cfec912a6483eb099fd96
+oid sha256:936ff8c2b9400fce9a0de200ff3e63db8bb605bad9cf2e8b4dd6ed7e14861eac
 size 1344847720

tokenizer_config.json

@@ -1 +1 @@
-{"eos_token": "</s>", "unk_token": "<unk>", "pad_token": "[PAD]", "additional_special_tokens": [], "bos_token": "<s>", "cls_token": "[CLS]", "sep_token": "[SEP]", "mask_token": "[MASK]", "do_lower_case": true, "extra_ids": 0}
+{"eos_token": "</s>", "unk_token": "<unk>", "pad_token": "[PAD]", "extra_ids": 0, "additional_special_tokens": [], "sp_model_kwargs": {}, "bos_token": "<s>", "cls_token": "[CLS]", "sep_token": "[SEP]", "mask_token": "[MASK]", "do_lower_case": true, "tokenizer_class": "T5Tokenizer"}