# Empirical AUC in validation set when no TRUE zeroes

In a cross-validation setting (LASSO penalized logistic regression), I'm calculating AUC. However, I'm interested in the variability of these estimates over the folds (this will give me an indication of the stability of my model selection over the folds).

As such, I want to find the empirical AUC in each of the 10 validation sets, and then calculate the variance over them. This poses a problem, as sometimes a validation set only holds only observations that have true outcome 1 or only observations that have true outcome 0. I don't know of a way to calculate the AUC in such a setting.

What would be the sensible approach here?

• Ignore this 'fold' in the calculations regarding AUC
• Give it some value anyway, like 0.5
• Perhaps you can suggest a way of approximating the AUC in such cases (adding 1 fake observation of the other kind and assume its predicted probability is either 0, 0.5 or 1?)
• Don't try this variance over the folds idea.
• thanks for accepting my answer, but after the remark of @cardinal I do not think that it is worth it (although it might be a contribution to consider). I hoped you would clarify the point whether you work in a "statistical" environment with only small amounts of data (instead of mass amount plus skewed class distribution (which I assumed)). May I suggest to edit your question / extend it to hopefully get more attention ? – steffen Jun 28 '11 at 7:02
• @Steffen: this was only a minor part of a much bigger setting (these fits actually happen in the middle of an EM), and the between-folds comparison was only marginally (in the nonstatistical sense) interesting to me. I decided to only calculate the overall AUC (just using the per-fold predicted probabilities). I agree with @cardinal that the "stratification" has its pitfalls, but your answer was the best I got :-) Especially your first sentence is dead on. – Nick Sabbe Jun 28 '11 at 7:29

• I would be (extremely?) cautious about your first recommendation. First off, I would not call that "stratified" sampling in the traditional sense, but rather sampling conditional on the response. Many times, that can get you in serious trouble with regards to inference. One example is in permutation testing, where "balancing" the permutations by ensuring there are the same proportion of positive and negative cases results as in the full data set results in (at times, wildly) anticonservative $p$-values. I would not be surprised if doing this with crossvalidation leads to similar behavior. – cardinal Jun 24 '11 at 14:19