12
$\begingroup$

Background: There are some great questions/answers here on how to calibrate models which predict probabilities of an outcome happening. For example

  1. Brier score, and its decomposition into resolution, uncertainty and reliability.
  2. Calibration plots and isotonic regression.

These methods often require the use of a binning method on the predicted probabilities, so that the behaviour of the outcome (0, 1) is smoothed over the bin by taking the mean outcome.

Problem: However, I cannot find anything which instructs me on how to choose the bin width.

Question: How do I choose the optimal bin width?

Attempt: Two common bin widths in use seem to be:

  1. Equal width binning, e.g. 10 bins each covering 10% of the the interval [0, 1].
  2. Tukey's binning method discussed here.

But are these choices of the bins the most optimal if one were interested in finding intervals in the predicted probabilities that are most miscalibrated?

$\endgroup$
  • 1
    $\begingroup$ If the "1" outcome is rare it's worth considering dividing to bins with equal number of "1"s instead of equal number of samples. This can help with maintaining the discrimination (AUC) of the model after the calibration $\endgroup$ – ihadanny Feb 23 '17 at 12:57
4
$\begingroup$

Any statistical method that uses binning has ultimately been deemed obsolete. Continuous calibration curve estimation has been commonplace since the mid 1990s. Commonly used methods are loess (with outlier detection turned off), linear logistic calibration, and spline logistic calibration. I go into this in detail in my Regression Modeling Strategies book and course notes. See http://www.fharrell.com/p/blog-page.html. The R rms package makes smooth nonparametric calibration curves easy to get, either using an independent external sample or using the bootstrap on the original model development sample.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

In my experience binning is good for visualizing probability distributions, but it is usually a bad idea, if one wants to use if for statistical tests and/or parameter inference. Primarily because one immediately limits the precision by the bin width. Another common problem is when the variable is not bound, i.e. one has to introduce low and high cutoffs.

Working with cumulative distributions in Kolmogorov-Smirnov spirit circumvents many of these problems. There are also many good statistical methods available in this case. (see, e.g., https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.