aee6f9e 2c7a271
1
2
3
# Japanese Dummy Tokenizer Repository containing a dummy Japanese Tokenizer trained on ```snow_simplified_japanese_corpus``` dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner.
