I am building a classifier and have over 1 million features to choose from. I implemented penalized regression, aka, lasso regression, followed by recursive feature selection in order to select features in each fold of my cross-validation. I only have 58 samples total.
When I run leave-one-out (LOO) cross-validation (cv), I get ROC AUCs around 0.9, but there is some variance each time I run it. When I ran 16x16-fold cv 100 times, the median AUC was only 0.6. The reason for choosing 16-fold is because that is the number of processors I have available to parallelize the code.
The lasso regression will only pick a total of n-1 features, where n is the number of samples in my training set. This means that for 16-fold cv, there are slightly fewer features that can be selected by lasso regression, so that is one possible explanation for the difference, but I am skeptical that is the reason for the 0.3 drop in AUC.
Any ideas on why it would perform so differently in 16-fold cv versus LOO cv? It should really only come down to a difference of training on 54 or 55 samples and predicting two or three vs. training on 57 and predicting 1 in each fold. Also, what does it mean that I am getting slightly different AUCs each time I run LOO cv?