Background: There are some great questions/answers here on how to calibrate models which predict probabilities of an outcome happening. For example
- Brier score, and its decomposition into resolution, uncertainty and reliability.
- Calibration plots and isotonic regression.
These methods often require the use of a binning method on the predicted probabilities, so that the behaviour of the outcome (0, 1) is smoothed over the bin by taking the mean outcome.
Problem: However, I cannot find anything which instructs me on how to choose the bin width.
Question: How do I choose the optimal bin width?
Attempt: Two common bin widths in use seem to be:
- Equal width binning, e.g. 10 bins each covering 10% of the the interval [0, 1].
- Tukey's binning method discussed here.
But are these choices of the bins the most optimal if one were interested in finding intervals in the predicted probabilities that are most miscalibrated?
