llm-jp/hf-slow-tokenizer-v21b1

Usage: (you need to specify use_fast=False, trust_remote_code=True as the args of AutoTokenizer.from_pretrained())

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "llm-jp/hf-slow-tokenizer-v21b1",
    legacy=True,
    use_fast=False,
    trust_remote_code=True,
)

When you create the tokenizer instance from the downloaded model with trust_remote_code=True, the model directory must be placed directly under the current directory, and the pretrained_model_name_or_path should not start with "./" but should end with "/".

tokenizer = AutoTokenizer.from_pretrained(
    "hf-slow-tokenizer-v21b1/",
    legacy=True,
    use_fast=False,
    trust_remote_code=True,
)