Why do we need separate data for probability calibration?
Scikit learn documentation says:
The samples that are used to fit the calibrator should not be the same samples used to fit the classifier, as this would introduce bias. This is because performance of the classifier on its training data would be better than for novel data. Using the classifier output of training data to fit the calibrator would thus result in a biased calibrator that maps to probabilities closer to 0 and 1 than it should.
Can someone provide more details? I can't decide if I want a biased calibrator trained on a massive amount of data or if I want an un-biased calibrator trained on little data.