Is there overfitting in my modelling approach despite cross-vaidation? My model is predicting a binomial dependent variable with a rich feature space of 20,000 independent variables. I am using the penalized logistic regression from the glmnet package, which works for the kind of dataset I have. I am using 10-fold cross-validation and the cross validation error is just 2.4%. I was told that with such a small cross validation error (in other words a very high accuracy of 97.6%) there is a high chance that my model is overfitting. Is is true?
If yes, then what is an alternative approach to this?
Here is some code which I am using:
library('glmnet')
data <- read.csv('datafile.csv', header=T)
mat = as.matrix(data)
X = mat[,1:ncol(mat)-1] 
y = mat[,ncol(mat)]
fit <- cv.glmnet(X,y, family="binomial", type.measure = "class")
betacoeff = as.matrix(fit$glmnet.fit$beta[,ncol(fit$glmnet.fit$beta)])

 A: Is there a large imbalance in the class labels in the data? If the dependent variable has an unequal distribution then an accuracy rate of 97.6% isn't necessary high. For example if 95% of your data contains a class label of 0, it's trivial to get 95% accuracy just  by always predicting a a class label of 0.
Assuming that this isn't the case, then 10-fold cross validation should give a reasonable prediction for the accuracy on unseen test data. That is, you probably aren't over-fitting in the usual sense.
If the 97.6% accuracy rate seems unreasonably high then it might be worth investigating some of your explanatory variables. Occasionally data sets contain "leaks from the future", that is, variables that provide information that normally wouldn't be available when making predictions. For example, data that only is filled in after the class label is already known.
A: Are you using all 20,000 variables in the model?  If so, then its very likely that a much smaller set will give nearly the same accuracy.  Investigating the single variable correlations to the outcome would also help you identify any 'leaks from the future' the previous poster mentioned.
You should also re-visit how the data set is created.  Have you thrown away too much "noisy" data which then leaves a less variable subset of the true data to fit?
Lastly, when using penalized regression the fitted coefficients in each fold are different, and some are zero in some models and non-zero in others. In effect you have 10 models, not 1.  Have you taken this into consideration and reported accuracy for a single model, or the overall accuracy for the models in each fold? 
