Tokenization bugs
#4
by
noam-converge
- opened
I noticed that there are a number of issues with the tokenization of this model; atoms with brackets [] and +n (ions) just return to their original atom, e.g. [Cl+2] returns to Cl, etc. In addition, the Brom atom (Br) is decoded to the B (boron) token, and more.
I also encounter the same issue.