In a cross-validation setting (LASSO penalized logistic regression), I'm calculating AUC. However, I'm interested in the variability of these estimates over the folds (this will give me an indication of the stability of my model selection over the folds).
As such, I want to find the empirical AUC in each of the 10 validation sets, and then calculate the variance over them. This poses a problem, as sometimes a validation set only holds only observations that have true outcome 1 or only observations that have true outcome 0. I don't know of a way to calculate the AUC in such a setting.
What would be the sensible approach here?
- Ignore this 'fold' in the calculations regarding AUC
- Give it some value anyway, like 0.5
- Perhaps you can suggest a way of approximating the AUC in such cases (adding 1 fake observation of the other kind and assume its predicted probability is either 0, 0.5 or 1?)
- Don't try this variance over the folds idea.