Hugging Face's logo

Spaces:
open-llm-leaderboard
/
open_llm_leaderboard
Running on CPU Upgrade

App Files Community
1079

All Orca Mini v* 70B Models Evaluation Failing

#1076
by pankajmathur - opened
Discussion
pankajmathur
•
edited 10 days ago

Hi Open LLM LB Team,

Thanks for all the help so far, I am not sure what is causing this, but there seems to be pattern in failure for all Orca Mini v* 70B models.

I have submitted these models as recently yesterday and as far as more then a month ago but every-time after successful submission there models are failing.

I am opening the discussion as needed, but is there any pattern of these failures, such as bigger param size like 70B.

Here is recent Orca Mini V9_4 70B failed request

https://huggingface.co/datasets/open-llm-leaderboard/requests/commit/30b4d9f396d34d9f0638979416eba11de2975eec

Could you please resubmit.

Thanks again for Open LLM LB

Regards,
Pankaj

pankajmathur

Another one failed, Here is recent Orca Mini V9_2 70B failed request :

https://huggingface.co/datasets/open-llm-leaderboard/requests/commit/8e4ccd0dd8b49c6037b5469fc39fb182fb345c66

pankajmathur

After almost 1 month of running time, orca_mini_v8_1_70b model failed too.....
https://huggingface.co/datasets/open-llm-leaderboard/requests/commit/115ed9127755826299401395ac630ae684173ccf

pankajmathur

Hi @alozowski @clefourrier ,

Hope you don’t mind tagging you on this issue, This evaluation queue almost took a month for orca_mini_v8_1_70b

https://huggingface.co/datasets/open-llm-leaderboard/requests/commit/115ed9127755826299401395ac630ae684173ccf

And then it just failed. Could you please advise on next steps?

Regards,
Pankaj

pankajmathur

Hi Team,

Please any update on this.

pankajmathur

Hi Team,

Please any update on this, it took more then a month for evaluation to run and then it just failed.

https://huggingface.co/datasets/open-llm-leaderboard/requests/commit/115ed9127755826299401395ac630ae684173ccf

Sign up or log in to comment