Spaces:

jordyvl

/

ece

like 3

Runtime error

Hi Leandro! Sorry to not respond sooner!
(On a personal note, I was hospitalized last week for surgery and needed some time to recover.)
Anyway, I was planning to continue this Sunday, July 3 the last edits.

• remake app.py for consistency with other projects
• import parsing of README and metric description
• add default example (e.g., predictions from BERT on some standard multi-class text classification dataset)
• [ISSUE] add default plot
• add test cases:
  • perfect accuracy - perfect calibration: ECE = 0
  • perfect accuracy - miscalibration: ECE > 0
  • imperfect accuracy - whichever calibration; separate bins and grouped bins: ECE > 0
• test cases on reliability diagram
• finish missing documentation

I also had a question regarding math notation in the metric description README; is this supported in some way?
Currently, this does not render properly: (in the meanwhile found the issue tracker at Gradio: https://github.com/gradio-app/gradio/issues/538) :)

$$EC{E}_p(f{)}^p={\mathbb{E}}_{(X,Y)}[\mathrm{\parallel }\mathbb{E}[Y=\widehat{y}\mid f(X)=\widehat{p}]-f(X){\mathrm{\parallel }}_p^p]ECE\_p(f)^p=\; \backslash mathbb\{E\}\_\{(X,Y)\}\; \backslash left[\backslash |\backslash mathbb\{E\}[Y\; =\; \backslash hat\{y\}\; \backslash mid\; f(X)\; =\; \backslash hat\{p\}]\; -\; f(X)\backslash |^p\_p\backslash right]$$, where $\hat{y} = \argmax_{y'}[f(X)]_y'$ is a class prediction with associated posterior probability $\hat{p}= \max_{y'}[f(X)]_y'$.

Sorry to hear! I hope you are feeling better and recovering fast! I didn't meant to put any pressure - I was just wondering if you hit any roadblocks that I could help get rid of :)

It's currently not supported in gradio (see this issue). As a workaround you could upload an image of the formula and embed it in the markdown.

Hi @lvwerra , made some updates following your suggestion with actions ticked as you can check above :)
Now the app looks like this for some sample inputs:

There are still some issues remaining, mainly with regards to how Gradio allows for data input.

• There seems to be an issue with how you can add a default value to a plot https://github.com/gradio-app/gradio/pull/1600
• In general, gradio is not very user-friendly to input np.arrays with a specific type; it has a tendency to convert everything to a string
• Do you have any suggestions on how to serve existing model probabilities and their (indexed) ground truth? The examplesargument can also be a directory, but then I am not sure how to proceed.

Thanks again for your "last mile" message, it encouraged me to tie up some loose ends :)

Hi, @lvwerra can you come back on some of my questions? I would like to finish this final loose ends :)

Hi @jordyvl , thanks for the ping.

There seems to be an issue with how you can add a default value to a plot https://github.com/gradio-app/gradio/pull/1600

If you need this change you can install gradio from the main branch by specifying it in the requirements.txt

In general, gradio is not very user-friendly to input np.arrays with a specific type; it has a tendency to convert everything to a string

The way I handle it in launch_gradio_widget is that if the input is a more complex structure than just a value (str, int, float) then interpret the input as a json string and load with json (see here).

Do you have any suggestions on how to serve existing model probabilities and their (indexed) ground truth? The examplesargument can also be a directory, but then I am not sure how to proceed.

Maybe an easy way would be to show them statically in the README?