1
$\begingroup$

I have several trained ML models (RandomForest, XGBoost and similar) for binary classification. Those models predict a score between 0 and 1 for different (but related) tasks. I would like to compare the scores to see which model yields a higher probability. Because the scores are not probabilities and therefore, are not directly comparable across models, I have to calibrate the scores before comparing them. I am doing it following classical procedures (e.g. a logistic regression on the target vs the score).

My question is: given that my goal is to compare those models, is it possible (is it a good idea?) to train only one calibration model on the union of all the scores given by all the models, instead of calibrating each model individually? I was thinking of including a categorical variable in the calibration model that says which model has generated that score, hence the calibration model would have two input features (score and origin) and a target variable that would be the true class (0 or 1) of the example. Do you know specific bibliography on that?

$\endgroup$
2

0