I have created two separate binary classifiers that predict the same kind of label using 2 separate datasets. The data is in the same format. They both have a AUC of 0.94 and 0.95

I have then created another binary classifier that combines the datasets of the above two models in order to predict the same label hoping for more accurate results. I have added an extra column to indicate the original dataset (0=coming from model 1, 1=coming from model 2) The AUC of the combined model is 0.943 but if I score the combined model on the two separate original dataset I get a 0.9 and 0.91. How does it come that the AUC of disjoint subsets is lower than the combined one?

It seems that in theory I could have a AUC of let's say 0.96 which is better than the older AUCs but then after splitting the data and scoring separately the sub AUCs could be worse. Which model is better at that point?

Has anybody come across to this problem ? Any good ways of dealing with it?

Thanks, Anton


I have never come across that particular problem, but maybe similar ones. My suggestion is that you are likely over-fitting your models, which might happen if you are doing a global fitting with your classifier and then testing in subsets of it (regardless if you used them for your training). If you have lots of features that is even more likely. To solve that you can try estimating a held-out data performance using k-fold cross validations, and report the median of your AUC in the k-testing sets. I hope this helps,

  • $\begingroup$ The AUCs are all evaluated on out of sample data, that should exclude overfitting? Training was done by early stopping on the out of sample AUCs ...the reported ones. $\endgroup$ – Anton Apr 22 '19 at 21:47
  • 1
    $\begingroup$ They should if nothing is overlapping and features are properly normalized in the whole dataset. Could you please state in a step-wise manner how the AUC values are generated? Importantly, if those are coming from an equal fraction of the data in both cases (e.g. 10%), and if they are an average of a k-fold procedure. $\endgroup$ – ilibarra Apr 22 '19 at 21:53
  • $\begingroup$ It's a big data set, my understanding is that K-fold validation is mostly useful for smaller datasets. Note that the two separate datasets don't have the same size. For separate AUCs and a fixed cut-point the ROC is simply TPR vs FPR but in this case TPR = (TPR0*NP0 + TPR1*NP1)/(NP0+NP1), FPR = (FPR0*NN0 + FPR1*NN1)/(NN0+NN1) where NP0 os the number of positives in set 0 and NN0 is the number of negatives in set 0, similarly for set 1. It seems more a properties of the maths? $\endgroup$ – Anton Apr 22 '19 at 22:02
  • $\begingroup$ If the two sets were balanced then it would be TPR = 0.5*(TPR0 + TPR1), FPR=0.5*(FPR0 + FPR1). But the two expressions are independent and symmetric as to where the FPRs and TPRs are cominig from. $\endgroup$ – Anton Apr 22 '19 at 22:04
  • $\begingroup$ K-fold purpose provides an unbiased performance metric so you are aware how models generalize. This applies to "small" and "big" datasets. My apologies but you must provide details on how the AUC values are calculated, otherwise I can do nothing more but speculate why this might happen. Maybe your out of sample data is biased, but I cannot tell. Your theoretical explanation seems reasonable, but it seems out of topic. I think that your Q requires knowing how your AUC values are calculated. If you provide more details on the data, it is also useful (e.g. nrows, nfeatures, pos/neg ratios). $\endgroup$ – ilibarra Apr 22 '19 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.