I've been scouring the web for more information on calibration curves. Scikit-learn has probably the best documentation I've found thus far. Here's their description:
When performing classification you often want not only to predict the class label, but also obtain a probability of the respective label. This probability gives you some kind of confidence on the prediction. Some models can give you poor estimates of the class probabilities and some even do not support probability prediction. The calibration module allows you to better calibrate the probabilities of a given model, or to add support for probability prediction.
Well calibrated classifiers are probabilistic classifiers for which the output of the predict_proba method can be directly interpreted as a confidence level. For instance, a well calibrated (binary) classifier should classify the samples such that among the samples to which it gave a predict_proba value close to 0.8, approximately 80% actually belong to the positive class.
Am I to understand if my model doesn't have a satisfactory calibration curve, then it is a bad model? If the top decile of my model's predicted probabilities has an actual event rate of 90%, but a mean predicted probability of 40%, is there anything inherently wrong with that outcome? I believe not, because my model is being used simply to rank individuals (and as such I've been using performance metrics like AUROC) – not interpret their actual probability of experiencing the event I'm interested in. Most probability calibration functions would not affect in a dramatic way my model's ability to rank, IIUC.