2
$\begingroup$

I always wondered how CalibratedClassifierCV was supposed to achieve probability calibration without a dedicated calibration set (which is appealing since no data is lost for training the classifier). Only when I looked at its documentation in detail I realized that what is called cross validation there is not really cross validation but rather an ensemble method, where simply k classifiers and k associated regressors are trained, whose average results are returned. Is that correct? And why is it not clearly stated in the documentation? I feel this greatly hampers understanding what this class actually does.

$\endgroup$
4
  • 1
    $\begingroup$ I think your points are spot on (+1); I had a look into the code of CalibratedClassifierCV.predict_proba(), and can confirm that it does an ensemble prediction and I completely agree that this should be clearly described in the documentation. But unless a developer of scikit-learn happens to come along here, we cannot be answer for the reasons behind this, nor can we address the your (justified) concerns. IMHO you should contact the developers of scikit-learn, so I'll vote to close this question. $\endgroup$ – cbeleites unhappy with SX Nov 10 '20 at 16:56
  • 2
    $\begingroup$ I’m voting to close this question because cross validated is not the place to answer questions about coding and documentation decisions in the scikit-learn project. This can only be addressed by the scikit-learn project (github.com/scikit-learn/scikit-learn/issues). $\endgroup$ – cbeleites unhappy with SX Nov 10 '20 at 16:59
  • 1
    $\begingroup$ Alright, thanks! I'm completely OK with closing the question given that there does not seem to be any widely accepted reason why the class is as it is. I might bring that up at a more fitting locality once I find the time. $\endgroup$ – JDsallin Nov 10 '20 at 17:51
  • $\begingroup$ As of version 0.24, there is a parameter ensemble that can override this behavior. $\endgroup$ – Ben Reiniger Mar 6 at 21:04