Should language token be mentioned with the input text ?
#13
by
arvariable
- opened
The document (https://huggingface.co/docs/transformers/model_doc/marian#multilingual-models) suggest that you should not use the language token for input text for this type of model (multi input, single output).
An example for language token for input text ">>fra<<où est l'arrêt de bus ?" would be ">>fra<<"
But without the language token in the input text , the output generated by the transformer is not accurate.
I am pretty new to this. As such, any advice on if language token is required or is this the point where more fine tuning of the model is the next step, is greatly appreciated.