2
$\begingroup$

I am fitting a logistic regression model on a data set with about 200,000 observation and 100 features. According to SAS output, the model converged correctly with an in-sample AUC of 0.85. However, when I applied the model on a few out of sample data set, the AUCs are as high as 0.98. This doesn't seem right to me. Usually the in-sample AUC is better than out-of-sample AUC. Any thoughts on the reason why this happened and how to interpret such results? I was using proc hplogistic in SAS 9.4.

Thanks everyone for your opinion. To be more specific, I was fitting the model using data in 2013, while I was testing the model on the data for 2014 Jan-June. Would that be causing this and how to adjust for that if needed? Besides, for response variable, the ratio of positive and negative is about 1:1.

$\endgroup$
1
  • $\begingroup$ This is somehow outdated, but maybe this would help. Have you used weights? SAS PROC LOGISTIC computes ROC incorrectly when you have weights with mean different than 1, you use /NORM option in WEIGHTS statement and ROCOPTIONS(WEIGHTED). Maybe this is the case. For me for example AUC could reach even values higher than 1 in such a situation, proving that error was involved. $\endgroup$
    – PtrZlnk
    May 2, 2017 at 8:53

2 Answers 2

3
$\begingroup$

You should keep in mind that the AUC is itself an estimator, so it has a variance. Notwithstanding a programming error, it's possible that 0.85 and 0.98 are statistically indistinguishable from each other.

If you're interested, here's a paper on computing confidence intervals for the AUC http://www.cs.nyu.edu/~mohri/pub/area.pdf so you conceivably could compute the confidence interval for the training AUC and the testing AUC and see if they overlap.

Unless you made a programming error, though, you're most likely fine since the AUC of the training and testing datasets are pretty close.

$\endgroup$
1
  • $\begingroup$ Or that they are statistically distinguishable, but that there is Type I error. $\endgroup$
    – Sycorax
    Sep 19, 2014 at 16:03
3
$\begingroup$

I agree to Blue Marker response. I'd just add:

  • The number of observations alone isn't that useful. I'd add the number of events so that the events per variable (EPV) can be estimated. Despite this it does appear that you do have enough observations and would be hard to argue you're over-fitting the model.
  • With EPV>>50, large data sizes, apparent AUC (the in-sample AUC) should approximate out-of-sample AUC.
  • In this case where they appear to be very different, you might be making an error in your out-of-sample validation. The easiest mistake is to use to0 small a validation dataset which will lead an unstable estimates. I don't think there are any fixed rules, but would want ~5,000 observations in out-of-sample set (Logistic Regression Model Validation)
  • Occasionally in the published literature with large datasets (>500k observations) the out-of-sample AUC will be larger, but then it is by 0.02 or a very small amount consistent with sampling error. The large difference you have implies an error in validation process.

Update

  • There are different forms of validation. Usually on this site one is referring to internal validation or simple external - where the out-of-sample data is assumed to come from the same distribution. But there is also temporal, geographic, full independent...validation where you're not just testing the model building process but also robustness against new data likely from a slightly different distribution. But this added level of validation should lead to a lower AUC, and does not explain the increased AUC.
  • How to find the source of error isn't clear to me. The easiest would be to reverse the training and test sets. Build a model of out-of-sample set, compare model with prior model, obtain in-sample and out-of-sample AUC. Since you have a fair amount of data, you could also alter the split to provide new out-of-sample set and rerun analysis.
  • You might want to check to make sure:
    (1) Your code is correct and giving you what you think it is giving you
    (2) There is nothing odd about the out-of-sample data such that isn't comparable to the in-sample data
    (3) That your AUC estimate is not sample size dependent, and you're not getting dramatically different results if chosen sample is changed or sample size changed.
$\endgroup$
1
  • $\begingroup$ Exactly, that's suspicious. Any other way that I can test my model? $\endgroup$ Sep 19, 2014 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.