My aim is to predict a binary output, given instances X and associated binary outputs y. I built a first classifier out-putting scores (not calibrated in probabilities). However, I identified in my dataset two subpopulations (different data generating processes, poor performance of the model on one of them). So I splitted my model in two, and refined my new models independently. The main refinement has been adding specific variables to both subpopulations.

Would it make sense to merge the scores given by my models ? Do models need to be calibrated in probabilities before being merged or could calibration in probabilities be achieved after ?


If the impetus behind your two models was two different subpopulations, then it sounds like the most natural course of action would be to apply the appropriate model to each sample, depending on what subpopulation the sample belongs to. No combination required.

If you do want to combine, I don't see a clear reason to prefer one over the other of your approaches. Try both.

  • Combine your scores (how exactly will depend on the semantics of your scores), then transform to probabilistic predictions, e.g., using a simple logistic regression, possibly spline-transforming your scores.
  • Extract probabilistic predictions from your two separate scores. E.g., run a logistic regression using the two scores (possibly spline-transformed, again) as two separate predictors.

Then see which approach performs better on a holdout set.

Of course, the first approach presupposes that your two models and their scores are even comparable. If this is not given, then the second approach is probably the only reasonable one.

  • $\begingroup$ Is there any rule of thumb to know if standard ML algorithms (generalised linear models, Neural Nets, xgboost trees...) results are comparable ? (in which sense ?) $\endgroup$ – lcrmorin Jul 18 '19 at 12:51
  • $\begingroup$ That depends on your model, and specifically what the "scores (not calibrated in probabilities)" are that you have. $\endgroup$ – Stephan Kolassa Jul 18 '19 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.