# Performance measure for calibration for binary classification problems?

Consider a binary classification problem, where the goal is to use training data $(x_i,y_i)_{i=1}^n$ to fit a classifier $f: \mathbb{R}^d \rightarrow [0,1]$ that outputs a conditional probability estimate (e.g. $f$ could be a logistic regression model).

The general way to check if the predicted probabilities match the true probabilities (i.e., are well-calibrated") seems to be a reliability plot. This plots the probabilities on the x-axis, and the observed probabilities on the y-axis.

I am looking for a performance metric that could be used instead of the reliability plot? Ideally, I'd like to find a metric that is used in the statistics or ML literature.

I ended up finding several measures in the literature (see e.g. CAL and MXE in the the paper Data Mining in Metric Space: An Empirical Analysis of Supervised Learning Performance Criteria by Caruana and Niculescu Mizil).

The most useful measure appears to be the Mean Calibration Error (CAL), which is the weighted root-mean squared error (RMSE) between predicted probabilities and true probabilities on a calibration plot. Formally:

$$\text{CAL} = \frac{1}{N} \sqrt{\sum_{k=1}^K \sum_{i \in B_k} {(p_k - \hat{p}_i})^2}$$

where:

• $\hat{p}_i$ is the predicted probability for example $i = 1,\ldots,N$
• $p_k$ is the observed probability for examples in bin $k$

Here, the binning is required because we do not typically have a "true" probability for each example $p_i$, only a label $y_i$. Thus, we construct $K$ bins (e.g., $B_1 = [0,0.1)$, $B_2 = [0.1,0.2)$...), and then estimate the observed probabilities for each bin as:

$$\hat{p}_k = \frac{1}{|B_k|}\sum_{i\in B_k} 1[y_i=1]$$

CAL is an intuitive summary statistic, but it does have several shortcomings. In particular:

• Since CAL is weighted by the number of observations, CAL can fudge local calibration issues over the full reliability diagram. If, for instance, 95% of your observations could fall into the first bin $\hat{p}_i \in [0,0.05)$ where you predict well... However, you could be completely off in the remaining cases.

• CAL depends on the binning procedure. This is why some people use a smoothed estimate (e.g., Caruana and Niculescu Mizil). This is not true in settings where classifiers output a discrete set of predictions (e.g., for risk scores)

• Wouldn't a model that just predicts the average probability for every data point get a near-perfect CAL score? Commented Aug 1, 2019 at 14:41
• It does! I couldn't find a better summary statistic for calibration, but I did come up with some best practices that I'll add here. In short: (1) always report CAL & AUC -- models that predict near the average probability for all points typically don't rank well so in cases like these the model will have low AUC. (2) Don't use CAL as a model selection metric (e.g., choosing a model that optimizes K-CV CAL tends to favor models that perform badly). Commented Aug 31, 2019 at 21:57
• Where did you take that definition of CAL? In the paper "Data Mining in Metric Space" it is defined using absolute differences instead of squares. Commented Sep 13, 2019 at 10:02
• Note that CAL and related calibration metrics are all sample size-biased: metric values will tend to be larger on smaller samples, given an equally well-calibrated model. (Especially problematic if comparing calibration between samples of different sizes.) See e.g. proceedings.mlr.press/v151/roelofs22a.html and arxiv.org/pdf/2302.08851.pdf. Disclaimer, I'm the first author on the latter one. We also propose a simple-to-use and unbiased calibration error metric. Commented Sep 6, 2023 at 21:55