I have implemented nested cross validation in Matlab for a classification problem. I have 56 features and 408 cases. I am performing feature and model selection in the inner cross-validation loop, using 10 fold CV for both inner and outer loops.
In the inner CV loop I employ a sequential forward feature selection procedure, nested inside a grid search to determine the optimum regularization parameters for a given regularized discriminant classifier model and selected feature set. I am finding very poor performance on the outer loop CV (when compared to a standard quadratic discriminant classifier, obtained with single loop CV), leading me to conclude I am overfitting in the inner CV loop.
I have read papers by Cawley and Talbot that show that biased model selection protocols favour worse models, and furthermore, that the inner loop procedure alone produces a biased performance estimate.
Does this explain the overfitting observed for the inner loop in nested CV?
Are there any practical strategies to avoid this overfitting?