How is confidence defined in Expected Calibration Error?

Question

I'm building a Bayesian Neural Network, and am trying to understand how to calibrate the uncertainty estimates. From a paper by Seedat and Kanan (https://chriskanan.com/wp-content/uploads/seedat2019.pdf) they define Expected Calibration Error (ECE) as follows: $$ ECE=\mathbb{E}[|\mathbb{P}(\hat{Y}=Y|\hat{P}=p)-p|] =\sum_{m=1}^m\frac{|B_m|}{n}|acc(B_m)-conf(B_m)| $$

where $m$=number of samples in bin, $M$=number of bins, $acc$=average accuracy for bin $B_m$, and $conf$ = confidence for bin $B_m$.

I understand everything here except what exactly the measure $conf$ actually is. Does it involve the confidence intervals for the predictions?

Answer 1

I believe that $conf(B_m)$ refers to the average confidence of the network at the bin $B_m$.

In order to implement the ECE, you set up M bins on the range=[0,1] with M<k where k is the amount of classes in your problem. Each data instance will be associated with the bin that corresponds to the largest output confidence value for that instance. After associating each data item with a bin, the $conf(B_m)$ is found by averaging the confidences in that bin. So for example, say in $B_0 = [0,0.25]$ you have 3 data items each with individual confidences 0.1,0.15, and 0.2. You would find $conf(B_0)$ by taking the average of the confidences in this bin, $$conf(B_0) = (0.1+0.15+0.2)/3 = 0.15$$.

I referenced this webpage by James D. McCaffrey, and he has an even more thorough explanation I would recommend looking into. https://jamesmccaffrey.wordpress.com/2021/01/22/how-to-calculate-expected-calibration-error-for-multi-class-classification/#:~:text=Expected%20calibration%20error%20(ECE)%20is,probabilities%20of%20a%20correct%20prediction