I'm sorry if the title is too vague. I'm not really sure of what I ask, this is a somewhat speculative question...
The setting is that I'm using XGBoost in a binary classification problem (40% positive class) , and I want to get predicted probabilities, so after the model is trained, I train an isotonic calibrier. What I get is the following:
- The first model, before calibration, was good enough in terms of AUC, obtaining 0,82.
- The probabilities were not calibried at all.
- After calibration, the probabilities are well calibrated, but I plotted the histogram of predicted probability and I got this:
So, after seeing this histogram, I began to think that the model was not very informative, or too conservative. I mean, let's imagine that we had all possible models with perfect calibration in front of us. Which one will be the best? And the worst? I think the worst will be the model that predicts the base rate for all instances, as this is the bare minimum that a non-informative but calibrated model can give. And the best will be the model with two peaks, on 0 and 1, a model that "knows" for certain the class of each instance. In between, the rest of models could be ordered in terms of how concentred is the histogram around the mean.
I'm not sure of what I'm saying, so I ask just in case there is a theory that supports this intuitions:
- Is it true that, among calibrated models, the more "disperse" the better? For example, the model with the bigger variance the better?
- And if so, if you start with a model not calibrated, and then calibrate it, will there be a relationship between the overall predictive power of the first model (in terms of AUC for example) and the variance of the calibrated model? Like some kind of constraint...
