File size: 440 Bytes
5295481 2d1a93e 5295481 2d1a93e 5295481 2d1a93e 5295481 2d1a93e 5295481 2d1a93e 5295481 2d1a93e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
---
library_name: transformers
license: mit
datasets:
- JeanKaddour/minipile
language:
- en
---
# BEE-spoke-data/MiniTokenizer-20480
This is a `ByteLevelBPETokenizer` trained on the `JeanKaddour/minipile` dataset with the aim to create a compact English-only tokenizer.
## Usage
load with AutoTokenizer, i.e.:
```py
from transformers import AutoTokenizer
tk = AutoTokenizer.from_pretrained('BEE-spoke-data/MiniTokenizer-20480')
tk
```
|