Spaces:
Running
on
CPU Upgrade
Any way to calculate/limit the cost?
I tried the demo version. This is an awesome tool! We are looking forward to start using it for our H4rmony Project, but our budget is very limited at the moment. I would like to know if there is any way to calculate and/or manage the cost.
Thank you!
Jorge
The easiest would be to duplicate the advanced space, and change the models to use smaller ones (though results might be less good) - you can also set a billing limit for the org you connect the token to. For the overall cost, it will depend on model used and number of samples generated.
cc
@sumuks
maybe we could add a small compute cost estimator to the github? wdyt?
@neovalle we're super excited that you liked it! there are a number of ways you can reduce / manage costs.
- pick the pareto frontier of models, based on your budget. you can find these in our paper here: https://arxiv.org/pdf/2504.01833 . i'm glad to help out in any way i can for this selection
- browse through the various config options here: https://github.com/huggingface/yourbench/blob/main/docs/CONFIGURATION.md
you can use a combination of cheaper models, and randomized question generation. you can set it to generate questions from a random percentage of the documents (say 5%) instead of generating from all. this will reduce your cost, but based on how dense your documents are, may reduce coverage of important content. you can also have a strict upper limit on cost, by setting it to generate questions from a maximum number of chunks!
do let us know if you need any help with this!
Thank you Clementine and Sumuk for your prompt response. This is great info, I'll try it with different configurations to find the best cost/benefit approach.