Cause of Very low AUC during 10-fold Cross-Validation I'm using 10-fold cross-validation (CV) with $L1$-penalized Logistic Regression to estimate expected prediction error. 
Briefly, each sample is part of the prediction set for exactly one fold, and I use this prediction compared to the ground truth of that sample, over all samples, to estimate the AUC below. I think this is relatively standard practice. 
$N=28$, $p=50$. Each feature is a real value in the interval $[0,2]$. 
The AUC that I estimate via this is really poor to the extent that the reversed classifier would be useful. Shown is for one regularization parameter $C$ for sklearn's implementation of Logistic Regression, but this effect is relatively robust across parameters (I tried $C=1 \times 10^{-5}$ up to $C=1 \times 10^{26}$), type of penalty ($L2$ vs. $L1$, Elastic Net, etc.), $k$ for $k$-fold CV, and even type of model (I also tried Support Vector Classifiers and Linear Discrimant Anslysis). 
My question is the following: beyond a simple coding bug, does anyone have any ideas what properties of the data and model(s) would lead to this behavior? 
Thanks for your thoughts. 

 A: 1) I expect what's actually going on is related to a pessimistic bias in cross validation estimates of roc auc that can arise in certain situations described here:
http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-8-326
In particular, they describe how it isn't actually valid to pool the predictions across the 10 folds as you are doing as they aren't samples from the same population. You need to calculate them separately and then combine. 
With so few data points you are likely also be getting issues with the base rate of class labels varying across the folds contributing to the effect. Stratified cross validation can help with this but only so much.
2) The phenomena of the inverse of a classifier doing better has been called "anti-learning" and some work has been done on it:
http://mark.reid.name/blog/anti-learning.html
It's usually related to having a data set where proximity implies dissimilar outcome.
A: I can see two obvious causes for this low AUC:


*

*The labels are getting reversed. Somehow either the logistic regression or ROC analysis thinks that 0 is 1 or 1 is 0. This is the most likely explanation.

*Overfitting. The fact that you have p > N is a sign that it is likely to happen, even with a "simple" model like logistic regression.
