# How to choose optimal bin width while calibrating probability models?

Background: There are some great questions/answers here on how to calibrate models which predict probabilities of an outcome happening. For example

1. Brier score, and its decomposition into resolution, uncertainty and reliability.
2. Calibration plots and isotonic regression.

These methods often require the use of a binning method on the predicted probabilities, so that the behaviour of the outcome (0, 1) is smoothed over the bin by taking the mean outcome.

Problem: However, I cannot find anything which instructs me on how to choose the bin width.

Question: How do I choose the optimal bin width?

Attempt: Two common bin widths in use seem to be:

1. Equal width binning, e.g. 10 bins each covering 10% of the the interval [0, 1].
2. Tukey's binning method discussed here.

But are these choices of the bins the most optimal if one were interested in finding intervals in the predicted probabilities that are most miscalibrated?

• If the "1" outcome is rare it's worth considering dividing to bins with equal number of "1"s instead of equal number of samples. This can help with maintaining the discrimination (AUC) of the model after the calibration Feb 23, 2017 at 12:57

Any statistical method that uses binning has ultimately been deemed obsolete. Continuous calibration curve estimation has been commonplace since the mid 1990s. Commonly used methods are loess (with outlier detection turned off), linear logistic calibration, and spline logistic calibration. I go into this in detail in my Regression Modeling Strategies book and course notes. See https://hbiostat.org/rms. The R rms package makes smooth nonparametric calibration curves easy to get, either using an independent external sample or using the bootstrap on the original model development sample.