Normalizing probabilities across different classification models

Question

Let's say I have a Logistic Regression model and Random Forest Model (multi-class) with different features selected for each model but from the same data source. I want to use the predict_proba() to get the predicted probabilities for each class.

The initial thought is that the probabilities are not comparable and that I would need to normalize or regularize the values in order to make them comparable. I'm not sure if I can use a probability calibrator: https://scikit-learn.org/stable/modules/calibration.html

Any thoughts on this? Is there another way to establish a confidence interval that is comparable across all models?

Answer 1

This does not make sense to me. Your models all predict probabilities. So might do a better job than others. The goal of your model comparison is to determine which models give the best predicted probabilities. If a model does a bad job of making such predictions, then it should be penalized for that poor performance, and the usual techniques like Brier score and log loss do exactly this.

You might find that one model makes poor probabilistic predictions but makes outstanding probabilistic predictions upon applying some calibration. In that case, you can compare the post-calibration predictions, considering the model to be the full pipeline of training the original model and then performing the calibration. Multi-class calibration, however, seems to be very-much an open question.