3
$\begingroup$

Why do we need separate data for probability calibration?

Scikit learn documentation says:

The samples that are used to fit the calibrator should not be the same samples used to fit the classifier, as this would introduce bias. This is because performance of the classifier on its training data would be better than for novel data. Using the classifier output of training data to fit the calibrator would thus result in a biased calibrator that maps to probabilities closer to 0 and 1 than it should.

Can someone provide more details? I can't decide if I want a biased calibrator trained on a massive amount of data or if I want an un-biased calibrator trained on little data.

$\endgroup$
1
  • $\begingroup$ You do not want a biased calibrator. Usually, you just randomly split your data. $\endgroup$
    – num_39
    Feb 13, 2023 at 9:31

1 Answer 1

4
$\begingroup$

If you first fit your model to some data and then calibrate it using the same data, you will follow the idiosyncrasies of this dataset too closely and overfit. It makes much more sense to split your training data into two parts: fit the model on one part and calibrate on the other. (Or use a method that gives you unbiased probabilistic classifications on the entire training sample.)

I can't decide if I want a biased calibrator trained on a massive amount of data or if I want an un-biased calibrator trained on little data.

If you have a biased model, it is systematically wrong. If you have trained this on a large amount of data, you will probably be very certain of your wrong model. This is not a good place to be. Better to have an unbiased model with more uncertainty.

$\endgroup$
6
  • $\begingroup$ What would be "a method that gives you unbiased probabilistic classifications on the entire training sample"? $\endgroup$
    – Eike P.
    Feb 13, 2023 at 10:04
  • $\begingroup$ @EikeP.: that's a good question, and I would say it hangs on the specifics of the case. Often calibration is done by simply feeding the output of the original model M into a logistic regression (possibly spline-transformed). In such a case, a "direct" model might be logistic regression trained on the inputs of M. $\endgroup$ Feb 13, 2023 at 10:32
  • $\begingroup$ ... and the reasoning would be that an LR model is typically low-dimensional and thus not prone to overfitting on the training dataset? Or why would you not want to calibrate the (likely multivariate) LR model fitted on the training set using another (univariante) LR model - or some other calibration model - on a holdout set? $\endgroup$
    – Eike P.
    Feb 13, 2023 at 15:00
  • $\begingroup$ @EikeP.: no, I am saying that using an appropriate logistic regression model should obviate the need for later calibration, so we can indeed use the entire training sample in the single modeling step. Alternatively, pick any other probabilistic classifier. $\endgroup$ Feb 13, 2023 at 15:19
  • 1
    $\begingroup$ @Glue: yes, you are right, I personally am a big fan of regularization. (1) If OP asks whether to run regularization across the same data as the original model was fitted on, I think we need to start with the simpler unbiased approach. This question is on a lower level than one where we can usefully discuss the bias-variance tradeoff or regularization. (2) OP mentions "massive" data, in which case I would rather be concerned about bias than variance. $\endgroup$ Feb 14, 2023 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.