Merging two model results My aim is to predict a binary output, given instances X and associated binary outputs y. I built a first classifier out-putting scores (not calibrated in probabilities). However, I identified in my dataset two subpopulations (different data generating processes, poor performance of the model on one of them). So I splitted my model in two, and refined my new models independently. The main refinement has been adding specific variables to both subpopulations.
Would it make sense to merge the scores given by my models ? Do models need to be calibrated in probabilities before being merged or could calibration in probabilities be achieved after ?
 A: If the impetus behind your two models was two different subpopulations, then it sounds like the most natural course of action would be to apply the appropriate model to each sample, depending on what subpopulation the sample belongs to. No combination required.
If you do want to combine, I don't see a clear reason to prefer one over the other of your approaches. Try both.


*

*Combine your scores (how exactly will depend on the semantics of your scores), then transform to probabilistic predictions, e.g., using a simple logistic regression, possibly spline-transforming your scores.

*Extract probabilistic predictions from your two separate scores. E.g., run a logistic regression using the two scores (possibly spline-transformed, again) as two separate predictors.


Then see which approach performs better on a holdout set.
Of course, the first approach presupposes that your two models and their scores are even comparable. If this is not given, then the second approach is probably the only reasonable one.
