On the subject of doing probabalistic classification and calibration with cross validation, the sklearn docs for Probability Calibration state:
CalibratedClassifierCVuses a cross-validation approach to fit both the classifier and the regressor. For each of the $k$ (trainset, testset) couple, a classifier is trained on the train set, and its predictions on the test set are used to fit a regressor. We end up with $k$ (classifier, regressor) couples where each regressor maps the output of its corresponding classifier into [0, 1]. Each couple is exposed in the
calibrated_classifiers_attribute, where each entry is a calibrated classifier with a
predict_probamethod that outputs calibrated probabilities. The output of
predict_probafor the main
CalibratedClassifierCVinstance corresponds to the average of the predicted probabilities of the $k$ estimators in the
In a setting where one is using multiple validation sets for model selection and model calibration, what is a principled way to combine the probability estimates produced by multiple probability estimators? As seen above,
sklearn averages the predicted probabilities, but is there any particular motivation for doing so? I couldn't find anything about this in a quick search through the literature. It seems to me like there are two broadly reasonable approaches:
- Choose and use the output of only one estimator. For example, pool the raw classifier outputs produced across the $k$ folds and use them to jointly train a single estimator. Or, somehow choose one of the estimators and discard the output of the others.
- Combine the estimator outputs in some way. Sklearn chooses this approach, combining the $k$ estimators produced during $k$-fold CV by averaging the probabilities. Other approaches could be taking the average of the parameters learned in Platt scaling, for example.
Is there research to support taking the mean as opposed to other alternatives for combining the multiple probabilistic classification estimators that arise during cross validation? Does this even matter much? (Empirically, I observe minimal difference between averaging the probability outputs of each estimator and averaging the coefficients of each estimator.)
Does anyone have any insight on this question, especially as far as justifying this decision? I'd like a better justification than "this is how sklearn does it", but I'm coming up short.