AI-Sweden-Models/gpt-sw3-40b marked with wrong metadata in the leaderboard

#421
by Ekgren - opened

The leaderboard shows the wrong metadata. TheAI-Sweden-Models/gpt-sw3-40b model is:

  • labeled as fine-tuned but the correct label should be pretrained
  • labeled as float16 but the correct label should be bfloat16

The model was added by hf staff when fixing a bug and that's the reason for the metadata difference. Hope the metadata can be updated!

Thank you for all your great work!

Open LLM Leaderboard org

Hi!
I can fix the fine-tuned/pretrained, but are you sure the model was submitted in bfloat16 and not in float16? The precision displayed is for the evaluation, not the model (weights are cast to the requested precision on the fly).

Hi!
I can fix the fine-tuned/pretrained, but are you sure the model was submitted in bfloat16 and not in float16? The precision displayed is for the evaluation, not the model (weights are cast to the requested precision on the fly).

Thank you and then the model was submitted in float16!

Thought it was metadata, the original model was trained in bfloat16 but results seem in line with expectation so probably worked out well in the end.

Open LLM Leaderboard org

Yep, there isn't a lot of difference between float16 and bfloat16 for most evaluations :)
Closing this issue, please wait for an hour for the modifs to be propagated to the leaderboard!

clefourrier changed discussion status to closed

Sign up or log in to comment