How to choose optimal bin width while calibrating probability models?

Question

Background: There are some great questions/answers here on how to calibrate models which predict probabilities of an outcome happening. For example

1. Brier score, and its decomposition into resolution, uncertainty and reliability.
2. Calibration plots and isotonic regression.

These methods often require the use of a binning method on the predicted probabilities, so that the behaviour of the outcome (0, 1) is smoothed over the bin by taking the mean outcome.

Problem: However, I cannot find anything which instructs me on how to choose the bin width.

Question: How do I choose the optimal bin width?

Attempt: Two common bin widths in use seem to be:

1. Equal width binning, e.g. 10 bins each covering 10% of the the interval [0, 1].
2. Tukey's binning method discussed here.

But are these choices of the bins the most optimal if one were interested in finding intervals in the predicted probabilities that are most miscalibrated?

Answer 1

Any statistical method that uses binning has ultimately been deemed obsolete. Continuous calibration curve estimation has been commonplace since the mid 1990s. Commonly used methods are loess (with outlier detection turned off), linear logistic calibration, and spline logistic calibration. I go into this in detail in my Regression Modeling Strategies book and course notes. See https://hbiostat.org/rms. The R rms package makes smooth nonparametric calibration curves easy to get, either using an independent external sample or using the bootstrap on the original model development sample.

Answer 2

In my experience binning is good for visualizing probability distributions, but it is usually a bad idea, if one wants to use if for statistical tests and/or parameter inference. Primarily because one immediately limits the precision by the bin width. Another common problem is when the variable is not bound, i.e. one has to introduce low and high cutoffs.

Working with cumulative distributions in Kolmogorov-Smirnov spirit circumvents many of these problems. There are also many good statistical methods available in this case. (see, e.g., https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test)

Answer 3

I agree with Frank Harrell, you shouldn't use methods which rely on binning. However, I'm more in favor of isotonic regression as the way to calibrate your probability forecasts. This does not require you to do binning and it is considered the sate of the art by some researchers. See for instance this paper for its advantages and more background.