2
$\begingroup$

I'm using 10-fold cross-validation (CV) with $L1$-penalized Logistic Regression to estimate expected prediction error.

Briefly, each sample is part of the prediction set for exactly one fold, and I use this prediction compared to the ground truth of that sample, over all samples, to estimate the AUC below. I think this is relatively standard practice.

$N=28$, $p=50$. Each feature is a real value in the interval $[0,2]$.

The AUC that I estimate via this is really poor to the extent that the reversed classifier would be useful. Shown is for one regularization parameter $C$ for sklearn's implementation of Logistic Regression, but this effect is relatively robust across parameters (I tried $C=1 \times 10^{-5}$ up to $C=1 \times 10^{26}$), type of penalty ($L2$ vs. $L1$, Elastic Net, etc.), $k$ for $k$-fold CV, and even type of model (I also tried Support Vector Classifiers and Linear Discrimant Anslysis).

My question is the following: beyond a simple coding bug, does anyone have any ideas what properties of the data and model(s) would lead to this behavior?

Thanks for your thoughts.

enter image description here

Improve this question
$\endgroup$
1
  • $\begingroup$ Update: very likely not to be a bug, as I've re-implemented this R's caret and also again in sklearn using a different implementation of CV (model_selection) and received similar (low AUC) results. $\endgroup$
    – ijoseph
    Dec 20, 2016 at 22:36

2 Answers 2

Reset to default
2
$\begingroup$

1) I expect what's actually going on is related to a pessimistic bias in cross validation estimates of roc auc that can arise in certain situations described here:

http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-8-326

In particular, they describe how it isn't actually valid to pool the predictions across the 10 folds as you are doing as they aren't samples from the same population. You need to calculate them separately and then combine.

With so few data points you are likely also be getting issues with the base rate of class labels varying across the folds contributing to the effect. Stratified cross validation can help with this but only so much.

2) The phenomena of the inverse of a classifier doing better has been called "anti-learning" and some work has been done on it:

http://mark.reid.name/blog/anti-learning.html

It's usually related to having a data set where proximity implies dissimilar outcome.

Improve this answer
$\endgroup$
2
  • $\begingroup$ "...data [that exhibits this pessimal AUC bias]... often has a small sample size and low discriminability between the classes e.g. in cancer prognostic and therapeutic response studies." Yes, this is exactly what I'm doing, so very likely related. The outcome is fairly unbalanced too (18 cases/ 10 controls), which is consistent with this effect. $\endgroup$
    – ijoseph
    Dec 20, 2016 at 22:43
  • $\begingroup$ I've been thinking about this, and for posterity, the best intuitive explanation is the following: for each CV fold, the fraction of true outcomes in the training set is negatively correlated the same fraction in the validation set. If no features are actually predictive, the classifier degenerates into just predicting whatever's outcome is more common in the training set, which will be consistently worse than guessing uniformly at random due to aforementioned anticorrelation. This can be, and was, somewhat addressed using stratified CV, but quantization prevents fully doing so. $\endgroup$
    – ijoseph
    Dec 22, 2016 at 0:52
1
$\begingroup$

I can see two obvious causes for this low AUC:

  1. The labels are getting reversed. Somehow either the logistic regression or ROC analysis thinks that 0 is 1 or 1 is 0. This is the most likely explanation.

  2. Overfitting. The fact that you have p > N is a sign that it is likely to happen, even with a "simple" model like logistic regression.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you. Re: the overfitting hypothesis, is it possible to overfit regardless of regularization parameter? I would have thought at some point greater than $C = 10^{-5}$ this would have happened for my relatively small (although $>1$) $p/N$ Ratio. Secondly, do you have intuition as to how overfitting would lead to a low AUC, rather than one close to $0.5$? (Sorry it won't let me edit my above comment so ignore it) $\endgroup$
    – ijoseph
    Dec 16, 2016 at 17:00
  • $\begingroup$ The regularization only makes overfitting harder, but it's not 100% foolproof and you can still think of weird structures in your data that can trick your regression to perform worse than random. $\endgroup$
    – Calimo
    Dec 16, 2016 at 17:06
  • $\begingroup$ Thanks again! Interesting. Do you have more intuition about what sort of weird structures would lead to this? For example, maybe a feature is perfectly predictive in the fitting portion of each CV fold, but is perfectly anti-predictive in the validating portion comes to mind. The follow up question would be how likely this would be in the event of real data. I suppose you've suggested that not likely, based on the bug hypothesis. (I still think there's value in speculating about this, though, to enhance my understanding of the underlying theory.) $\endgroup$
    – ijoseph
    Dec 19, 2016 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.