<|input|> Were these used as special tokens during training <|output|>
#15
by
aschroeder91
- opened
the <|input|> and <|output|> used in inference in the model code suggests that this was used during training the model, but looking at the tokenizer config files it looks like these are not special tokens but I am not 100% sure if this was true during training.
Should I add them as special tokens during fine-tuning this model? or leave it as is?
Hi, you can just leave them as is, they are not considered special tokens.
I also encourage you to use the newer model NuExtract-1.5, unless there is a specific reason you prefer this one.
oh no! i meant to put this on NuExtract-1.5 (which I am using)