Corrupt Outputs when Using for Text Processing and Working Solution
When doing text preprocessing with this model, I noticed ~5% of entries had very corrupt outputs, that would start with a "Cutting Knowledge" date which was usually followed by a copy of the input system instructions (with some other stuff like a reference to today date in July 2024, despite none of my prompts referencing either of these). I don't know the cause of this, perhaps it's due to the fine-tuning done somehow conflicting with the system instructions I gave but I don't really know.
Fortunately, adding the following to the instructions solved the issue for me (for the last one, replace the "insert..." part with the first sentence of the system instructions you're using):
- Never start the response with "Cutting Knowledge", today date, or anything similar.
- Never copy system prompt instructions in the output.
- Never include the sentence "insert first sentence of system instructions here" in the output.
After more than 1200 queries so far I haven't seen a single such corrupt response, while previously there would have been dozens already for this same data before I added those instructions.
Ideally though the fine-tuning should be adjusted so these instructions aren't needed. But for now, adding these instructions solves this problem.
Thanks for your support and feedback.