Models
Datasets
Spaces
Posts
Docs
Enterprise
Pricing
Log In
Sign Up

Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / vocab /gpt_nexo_20b

2 contributors

History: 3 commits

xu-song's picture

update

0ce6477 over 1 year ago

tokenizer
update over 1 year ago
tokenzier_hf
update over 1 year ago
20B_tokenizer.json

2.47 MB

update over 1 year ago
20B_tokenizer.zh.json

2.11 MB

update over 1 year ago
README.md

1.56 kB

update over 1 year ago
__init__.py

717 Bytes

update over 1 year ago
convert_vocab_to_txt.py

459 Bytes

update over 1 year ago
test_gpt_neox_20b.py

2.96 kB

update over 1 year ago
test_hf_gpt_neox.py

487 Bytes

update over 1 year ago
test_oov.py

505 Bytes

update over 1 year ago
test_special_token.py

340 Bytes

update over 1 year ago
test_tokenizer.py

1.94 kB

update over 1 year ago
test_tokenizer_HF.py

1.3 kB

update over 1 year ago
test_zh_coding_len.py

447 Bytes

update over 1 year ago
vocab.zh.txt

9.34 kB

update over 1 year ago