5
$\begingroup$

In a cross-validation setting (LASSO penalized logistic regression), I'm calculating AUC. However, I'm interested in the variability of these estimates over the folds (this will give me an indication of the stability of my model selection over the folds).

As such, I want to find the empirical AUC in each of the 10 validation sets, and then calculate the variance over them. This poses a problem, as sometimes a validation set only holds only observations that have true outcome 1 or only observations that have true outcome 0. I don't know of a way to calculate the AUC in such a setting.

What would be the sensible approach here?

  • Ignore this 'fold' in the calculations regarding AUC
  • Give it some value anyway, like 0.5
  • Perhaps you can suggest a way of approximating the AUC in such cases (adding 1 fake observation of the other kind and assume its predicted probability is either 0, 0.5 or 1?)
  • Don't try this variance over the folds idea.
|cite|improve this question
$\endgroup$
  • $\begingroup$ thanks for accepting my answer, but after the remark of @cardinal I do not think that it is worth it (although it might be a contribution to consider). I hoped you would clarify the point whether you work in a "statistical" environment with only small amounts of data (instead of mass amount plus skewed class distribution (which I assumed)). May I suggest to edit your question / extend it to hopefully get more attention ? $\endgroup$ – steffen Jun 28 '11 at 7:02
  • $\begingroup$ @Steffen: this was only a minor part of a much bigger setting (these fits actually happen in the middle of an EM), and the between-folds comparison was only marginally (in the nonstatistical sense) interesting to me. I decided to only calculate the overall AUC (just using the per-fold predicted probabilities). I agree with @cardinal that the "stratification" has its pitfalls, but your answer was the best I got :-) Especially your first sentence is dead on. $\endgroup$ – Nick Sabbe Jun 28 '11 at 7:29
2
$\begingroup$

Applying a model on a set with only positives or negatives does not allow you to make any statement about the differentiation power of the model. Hence I would ignore this folds.

I suggest to improve the whole validation process in the following way:

  • use a stratified sampling approach within cross-validation to ensure that the ratio of positives to negatives is the same in every fold
  • and/or repeat the crossvalidation some times with different seeds (at maximum 6 times, as suggested by Kohavi) to gain more folds to average over.
|cite|improve this answer
$\endgroup$
  • $\begingroup$ I would be (extremely?) cautious about your first recommendation. First off, I would not call that "stratified" sampling in the traditional sense, but rather sampling conditional on the response. Many times, that can get you in serious trouble with regards to inference. One example is in permutation testing, where "balancing" the permutations by ensuring there are the same proportion of positive and negative cases results as in the full data set results in (at times, wildly) anticonservative $p$-values. I would not be surprised if doing this with crossvalidation leads to similar behavior. $\endgroup$ – cardinal Jun 24 '11 at 14:19
  • $\begingroup$ @cardinal yes, you are correct. I am a ML and all the times I have encountered a situation like the OP I had a ton of data but a skewed response distribution (like 1% positives), so permutation tests are not feasible. In these situtations it is hard enough to stabilize a model (so that AUC does not drop below 0.5 in a single fold) even with sampling conditionally on the inference (at least from my experience, but I do not have much) $\endgroup$ – steffen Jun 24 '11 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.