Background: There are some great questions/answers here on how to calibrate models which predict probabilities of an outcome happening. For example

  1. Brier score, and its decomposition into resolution, uncertainty and reliability.
  2. Calibration plots and isotonic regression.

These methods often require the use of a binning method on the predicted probabilities, so that the behaviour of the outcome (0, 1) is smoothed over the bin by taking the mean outcome.

Problem: However, I cannot find anything which instructs me on how to choose the bin width.

Question: How do I choose the optimal bin width?

Attempt: Two common bin widths in use seem to be:

  1. Equal width binning, e.g. 10 bins each covering 10% of the the interval [0, 1].
  2. Tukey's binning method discussed here.

But are these choices of the bins the most optimal if one were interested in finding intervals in the predicted probabilities that are most miscalibrated?

  • 1
    $\begingroup$ If the "1" outcome is rare it's worth considering dividing to bins with equal number of "1"s instead of equal number of samples. This can help with maintaining the discrimination (AUC) of the model after the calibration $\endgroup$
    – ihadanny
    Feb 23, 2017 at 12:57

3 Answers 3


Any statistical method that uses binning has ultimately been deemed obsolete. Continuous calibration curve estimation has been commonplace since the mid 1990s. Commonly used methods are loess (with outlier detection turned off), linear logistic calibration, and spline logistic calibration. I go into this in detail in my Regression Modeling Strategies book and course notes. See https://hbiostat.org/rms. The R rms package makes smooth nonparametric calibration curves easy to get, either using an independent external sample or using the bootstrap on the original model development sample.


In my experience binning is good for visualizing probability distributions, but it is usually a bad idea, if one wants to use if for statistical tests and/or parameter inference. Primarily because one immediately limits the precision by the bin width. Another common problem is when the variable is not bound, i.e. one has to introduce low and high cutoffs.

Working with cumulative distributions in Kolmogorov-Smirnov spirit circumvents many of these problems. There are also many good statistical methods available in this case. (see, e.g., https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test)


I agree with Frank Harrell, you shouldn't use methods which rely on binning. However, I'm more in favor of isotonic regression as the way to calibrate your probability forecasts. This does not require you to do binning and it is considered the sate of the art by some researchers. See for instance this paper for its advantages and more background.


This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .