0
$\begingroup$

I have a classification problem for which I have a dataset composed by 89 istances (59 of class 0 and 30 of class 1).

Given the small dataset I perform a leave-one-out cross validation and then compute the AUC on the out-of-sample scores.

I get an AUC of 0.08.... I don't understand...should the AUC be more than 0.5?

$\endgroup$
  • $\begingroup$ The AUC does nt have to be > 0.5 on a test set, as your model could conceivably generalize so poorly as to be worse than random assignment. A 0.08 points to a programming error though, since you could reverse all the predictions of your model and have another with AUC 0.92, which is really good. Maybe you reversed your classes at some point accidentally? $\endgroup$ – Matthew Drury Feb 1 '18 at 16:11
  • $\begingroup$ no I don't think so.. labels are fine... I know I could reverse the labels but I dont feel good about that...especially in an academic work.. would that be accepted? $\endgroup$ – gabboshow Feb 1 '18 at 16:18
  • $\begingroup$ No, I think you do have to investigate what's going on. But the label reversal thing maybe gives you a place to start looking I suppose. $\endgroup$ – Matthew Drury Feb 1 '18 at 16:24
1
$\begingroup$

Particular classifiers can give consistently wrong leave-one-out predictions (typically with small sample sizes).

The reason is often that with LOO, the tested class is always underrepresented in the training set wrt the whole data set.

If this is the case, statified resampling validation (e.g. stratified k-fold cross validation) would be better. In your case, stratified 20-fold (= leave-3-out) or 10-fold would fit nicely with the data.


And no, reversing labels (without clear proof how they came to be confused - which means you correct a then-known error) is IMHO never a good idea.

What you can do, though: randomly mix the labels. This gives you a data set where you know that performance can only be guessing. If on that data you also find << 0.5 AUC for the LOO, that also hints at the "consistently wrong because systematically underrepresented" explanation for the LOO results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.