Add `"add_prefix_space": true,`; this allows for much stronger token-level performance (e.g. NER, ColBERT)
Browse filesHello!
## Pull Request overview
* Add `"add_prefix_space": true,` to the tokenizer config
## Details
This allows for much stronger token-level performance (e.g. NER, ColBERT), because otherwise each token will not be prepended by a space, while our model is trained with data where each token is prepended by a space.
We will need to explain that users can set `add_prefix_space` to False in the model card somewhere.
cc
@bclavie
@bwarner
@NohTow
could one of you take care of that?
P.s. feel free to hold off on merging for now, this PR can also be used to run some tests first (with `revision="refs/pr/..."`).
Note that you need to use `transformers` after https://github.com/huggingface/transformers/pull/35593 was merged.
- Tom Aarsen
- tokenizer_config.json +1 -0
tokenizer_config.json
CHANGED
@@ -1,4 +1,5 @@
|
|
1 |
{
|
|
|
2 |
"added_tokens_decoder": {
|
3 |
"0": {
|
4 |
"content": "|||IP_ADDRESS|||",
|
|
|
1 |
{
|
2 |
+
"add_prefix_space": true,
|
3 |
"added_tokens_decoder": {
|
4 |
"0": {
|
5 |
"content": "|||IP_ADDRESS|||",
|