hugohow
/

creole_reunion_tokenizer

Réunion Creole French

Inference Endpoints

Model card Files Files and versions Community

hugohow commited on 23 days ago

Commit

7c4d044

•

1 Parent(s): 45828fb

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -11,7 +11,7 @@ This tokenizer is specifically designed for working with **Réunion Creole**, a
 ## Features
 - Built using the **BPE (Byte Pair Encoding)** model.
-- Trained on "LA RIME, Mo i akorde dann bal zakor".
 - Supports special tokens for common NLP tasks:
   - `[CLS]`: Start-of-sequence token for classification tasks.
   - `[SEP]`: Separator token for multi-segment inputs.

 ## Features
 - Built using the **BPE (Byte Pair Encoding)** model.
+- Trained on "LA RIME, Mo i akorde dann bal zakor", a free-access book.
 - Supports special tokens for common NLP tasks:
   - `[CLS]`: Start-of-sequence token for classification tasks.
   - `[SEP]`: Separator token for multi-segment inputs.