zirui3
/

llm-multilingual-tokenizer

Model card Files Files and versions Community

llm-multilingual-tokenizer / README.md

zirui3's picture

Upload README.md

3315142 over 1 year ago

|

143 Bytes



	# summary
	multilingual tokenizer trained on multilingual data by using the SentencePiece library and the BPE algorithm.

	* vocab size: 100k