How to choose optimal bin width while calibrating probability models?

Background: There are some great questions/answers here on how to calibrate models which predict probabilities of an outcome happening. For example

1. Brier score, and its decomposition into resolution, uncertainty and reliability.
2. Calibration plots and isotonic regression.

These methods often require the use of a binning method on the predicted probabilities, so that the behaviour of the outcome (0, 1) is smoothed over the bin by taking the mean outcome.

Problem: However, I cannot find anything which instructs me on how to choose the bin width.

Question: How do I choose the optimal bin width?

Attempt: Two common bin widths in use seem to be:

1. Equal width binning, e.g. 10 bins each covering 10% of the the interval [0, 1].
2. Tukey's binning method discussed here.

But are these choices of the bins the most optimal if one were interested in finding intervals in the predicted probabilities that are most miscalibrated?

• If the "1" outcome is rare it's worth considering dividing to bins with equal number of "1"s instead of equal number of samples. This can help with maintaining the discrimination (AUC) of the model after the calibration Feb 23 '17 at 12:57