On the subject of doing probabalistic classification and calibration with cross validation, the sklearn docs for Probability Calibration state:

CalibratedClassifierCV uses a cross-validation approach to fit both the classifier and the regressor. For each of the $k$ (trainset, testset) couple, a classifier is trained on the train set, and its predictions on the test set are used to fit a regressor. We end up with $k$ (classifier, regressor) couples where each regressor maps the output of its corresponding classifier into [0, 1]. Each couple is exposed in the calibrated_classifiers_ attribute, where each entry is a calibrated classifier with a predict_proba method that outputs calibrated probabilities. The output of predict_proba for the main CalibratedClassifierCV instance corresponds to the average of the predicted probabilities of the $k$ estimators in the calibrated_classifiers_ list.

(Emphasis mine)

In a setting where one is using multiple validation sets for model selection and model calibration, what is a principled way to combine the probability estimates produced by multiple probability estimators? As seen above, sklearn averages the predicted probabilities, but is there any particular motivation for doing so? I couldn't find anything about this in a quick search through the literature. It seems to me like there are two broadly reasonable approaches:

  1. Choose and use the output of only one estimator. For example, pool the raw classifier outputs produced across the $k$ folds and use them to jointly train a single estimator. Or, somehow choose one of the estimators and discard the output of the others.
  2. Combine the estimator outputs in some way. Sklearn chooses this approach, combining the $k$ estimators produced during $k$-fold CV by averaging the probabilities. Other approaches could be taking the average of the parameters learned in Platt scaling, for example.

Is there research to support taking the mean as opposed to other alternatives for combining the multiple probabilistic classification estimators that arise during cross validation? Does this even matter much? (Empirically, I observe minimal difference between averaging the probability outputs of each estimator and averaging the coefficients of each estimator.)

Does anyone have any insight on this question, especially as far as justifying this decision? I'd like a better justification than "this is how sklearn does it", but I'm coming up short.

  • $\begingroup$ For future seekers, we observe no particular empirical difference between these options. My colleague pointed out that the sklearn approach actually differs from the cross-validation approach recommended by Platt in his original 1999 paper; Platt recommends pooling all validation predictions and fitting a single model on top of that, not fitting $k$ different models. $\endgroup$ – Suriname0 Jul 27 '20 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.