what does it mean when out of sample AUC is greater than in sample AUC?

Question

I am fitting a logistic regression model on a data set with about 200,000 observation and 100 features. According to SAS output, the model converged correctly with an in-sample AUC of 0.85. However, when I applied the model on a few out of sample data set, the AUCs are as high as 0.98. This doesn't seem right to me. Usually the in-sample AUC is better than out-of-sample AUC. Any thoughts on the reason why this happened and how to interpret such results? I was using proc hplogistic in SAS 9.4.

Thanks everyone for your opinion. To be more specific, I was fitting the model using data in 2013, while I was testing the model on the data for 2014 Jan-June. Would that be causing this and how to adjust for that if needed? Besides, for response variable, the ratio of positive and negative is about 1:1.

Answer 1

You should keep in mind that the AUC is itself an estimator, so it has a variance. Notwithstanding a programming error, it's possible that 0.85 and 0.98 are statistically indistinguishable from each other.

If you're interested, here's a paper on computing confidence intervals for the AUC http://www.cs.nyu.edu/~mohri/pub/area.pdf so you conceivably could compute the confidence interval for the training AUC and the testing AUC and see if they overlap.

Unless you made a programming error, though, you're most likely fine since the AUC of the training and testing datasets are pretty close.

Answer 2

I agree to Blue Marker response. I'd just add:

• The number of observations alone isn't that useful. I'd add the number of events so that the events per variable (EPV) can be estimated. Despite this it does appear that you do have enough observations and would be hard to argue you're over-fitting the model.
• With EPV>>50, large data sizes, apparent AUC (the in-sample AUC) should approximate out-of-sample AUC.
• In this case where they appear to be very different, you might be making an error in your out-of-sample validation. The easiest mistake is to use to0 small a validation dataset which will lead an unstable estimates. I don't think there are any fixed rules, but would want ~5,000 observations in out-of-sample set (Logistic Regression Model Validation)
• Occasionally in the published literature with large datasets (>500k observations) the out-of-sample AUC will be larger, but then it is by 0.02 or a very small amount consistent with sampling error. The large difference you have implies an error in validation process.

Update

• There are different forms of validation. Usually on this site one is referring to internal validation or simple external - where the out-of-sample data is assumed to come from the same distribution. But there is also temporal, geographic, full independent...validation where you're not just testing the model building process but also robustness against new data likely from a slightly different distribution. But this added level of validation should lead to a lower AUC, and does not explain the increased AUC.
• How to find the source of error isn't clear to me. The easiest would be to reverse the training and test sets. Build a model of out-of-sample set, compare model with prior model, obtain in-sample and out-of-sample AUC. Since you have a fair amount of data, you could also alter the split to provide new out-of-sample set and rerun analysis.
• You might want to check to make sure:
  (1) Your code is correct and giving you what you think it is giving you
  (2) There is nothing odd about the out-of-sample data such that isn't comparable to the in-sample data
  (3) That your AUC estimate is not sample size dependent, and you're not getting dramatically different results if chosen sample is changed or sample size changed.