6
$\begingroup$

From what I understood by reading sklearn Probability Calibration, when we run CalibratedClassifierCV we will fit "a regressor (called a calibrator) that maps the output of the classifier (as given by decision_function or predict_proba) to a calibrated probability in [0, 1]". However, its mentioned that we can use prefit or not and we could use ensemble or not, but both parameters aren't clear for me.

When we use cv=prefit, we will split the data into train, test and calibration sets, then fit a model using train sets, calibrate with calibration set and later use the calibrated model with the test set? Whhen we don't use cv=prefit, we use only train and test? And the same train set used to fit the model is used to calibrate?

What about ensemble = True? This parameter is not clear at all for me.

$\endgroup$

1 Answer 1

4
$\begingroup$

When we use cv=prefit, we will split the data into train, test and calibration sets, then fit a model using train sets, calibrate with calibration set and later use the calibrated model with the test set?

Yes, that's probably the usual approach.

When we don't use cv=prefit, we use only train and test? And the same train set used to fit the model is used to calibrate? [...]

Correct; that's the CV of CalibratedClassifierCV: the training set gets cross-validated over, with training folds training the classifier, then the left-out fold serving the purpose of training data for the calibrator (using predictions from the classifier).

[...] And the same train set used to fit the model is used to calibrate?

Yes, in the sense of the above. But note that it isn't ever the same model training on one set and then using its predictions on its own training set for calibrator training; that leads to biased data training the calibrator.

What about ensemble = True?

Let's say we're doing cv=3, i.e. 3-fold cross-validation in the calibrator training setting. We train three models, each on two folds, and they each make a prediction on the third fold. So we have one prediction for every point in our training set (again, it's important that each of these predictions are made by models that didn't train on that point). We can

  1. just throw all those predictions together to train a single calibrator (ensemble=False), or
  2. train separate calibrators on each fold, so that we actually have three (classifier, calibrator) pairs. Then we can make final predictions as a simple averaging ensemble of those pairs. (Each pair predicts using its base classifier's probability fed into its calibrator; we average the outputs of all those pairs to get a final output.)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.