Conversion of tokens >=32000
#1
by
LuckiestOne
- opened
Hey, thanks a lot for the quantized versions! I got a quick question though: how did you handle the tokens >= 32000? There are some special tokens in that range, did they get skipped during conversion or did you keep them?