I think the ckpt orders are messed up.
Hello, we are conducting an experiment to observe the perplexity values exhibited by SmolLM2. For this experiment, we randomly extracted 1,000 samples from the Cosmopedia-v2 dataset and measured the perplexity of intermediate checkpoints for SmolLM2: 135M, 360M, and 1.7B. However, during the experiment, we discovered an anomaly in the intermediate checkpoint of the 1.7B model, prompting us to report this issue.
The results showed that, as expected, the 135M and 360M models exhibited a stable decrease in perplexity over training steps. However, the 1.7B model started with an abnormally high perplexity value, which did not decrease and instead diverged over time. While the 1.7B checkpoint showed somewhat reasonable perplexity values, it failed to demonstrate a stable trend, suggesting that the issue lies with the model's checkpoint rather than the input language. To verify whether the same issue occurs in the final model, SmolLM2-1.7B, we measured its perplexity as well.
The results showed that, unlike the intermediate checkpoints, the final model demonstrated a significantly lower perplexity of 1.2524310472011566 compared to the previous checkpoints.
These experimental results suggest that there is indeed an issue with the intermediate checkpoint of the 1.7B model. We kindly request a review of this matter.
Hey! I've used the wrong script to convert the checkpoint from nanotron to huggingface, I re-uploaded the 1.7B checkpoint yesterday so should be good now! Sorry for the inconvenience, please tell me if it's ok with the new ckpt, thanks :)
Thank you, @eliebak ! I re-evaluated the perplexity for updated checkpoints, and they showed regular perplexities!
Perfect, closing this issue, thanks a lot :)