I have created two separate binary classifiers that predict the same kind of label using 2 separate datasets. The data is in the same format. They both have a AUC of 0.94 and 0.95
I have then created another binary classifier that combines the datasets of the above two models in order to predict the same label hoping for more accurate results. I have added an extra column to indicate the original dataset (0=coming from model 1, 1=coming from model 2) The AUC of the combined model is 0.943 but if I score the combined model on the two separate original dataset I get a 0.9 and 0.91. How does it come that the AUC of disjoint subsets is lower than the combined one?
It seems that in theory I could have a AUC of let's say 0.96 which is better than the older AUCs but then after splitting the data and scoring separately the sub AUCs could be worse. Which model is better at that point?
Has anybody come across to this problem ? Any good ways of dealing with it?