Addition of FPS to benchmark
I think something that's important to take into consideration is how fast a network is. It would be nice to have FPS added to the metrics (assuming the use of an H100 or V100)
Thank you for pointing that out.
You're absolutely right. Inference time is a crucial metric when comparing models. We'll certainly take that into consideration. 🤗
Column FPS has just been added! 🔥
I also added the following comment on the "📈 Metrics" tab:
"We measure the frames per second (FPS) for each model by looking at the average time it takes across the whole dataset. This includes the pre and post processing steps.
The hardware we use definitely plays a role in these numbers. You can see which hardware we used in the results table.
Because each model has its own specific needs when it comes to batch size and memory requirements, we decided to test them with just 1 image per batch. One thing to keep in mind: this test setup might not fully reflect real-world scenarios. Typically, more images are processed together to get things moving faster. "