No token for "the"?

by FremyCompany - opened Feb 28, 2023

Feb 28, 2023

I think there might be something strange going on with the tokenizer, because it does not have a token for "Ġthe" despite having a token for "ĠThe": 409.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment