4
$\begingroup$

I have implemented nested cross validation in Matlab for a classification problem. I have 56 features and 408 cases. I am performing feature and model selection in the inner cross-validation loop, using 10 fold CV for both inner and outer loops.

In the inner CV loop I employ a sequential forward feature selection procedure, nested inside a grid search to determine the optimum regularization parameters for a given regularized discriminant classifier model and selected feature set. I am finding very poor performance on the outer loop CV (when compared to a standard quadratic discriminant classifier, obtained with single loop CV), leading me to conclude I am overfitting in the inner CV loop.

I have read papers by Cawley and Talbot that show that biased model selection protocols favour worse models, and furthermore, that the inner loop procedure alone produces a biased performance estimate.

Does this explain the overfitting observed for the inner loop in nested CV?

Are there any practical strategies to avoid this overfitting?

$\endgroup$
2
$\begingroup$

Overfitting in model selection problems for classification is usually due to including too many parameters. Cross-validation or bootstrap error rate estimation should avoid this problem because it avoids the optimism of an estimate like resubstitution which tests the classifier on the same data used in the fit. If you minimize the cross-validated estimate of error rate in your inner loop as the criterion for variable selection you should not have this problem. Am I correct in assuming that your selection procedure does not do this? If so you are probably using a procedure that is biased toward models with many parameters that may be poor models for prediction.

$\endgroup$
  • $\begingroup$ I am minimizing the cross-validation error (maximizing classification accuracy) in the inner loop - I am choosing the model that produces the smallest error for a given set of features and regularization parameters. Generally the models produced are quite small in terms of features. $\endgroup$ – BGreene Sep 12 '12 at 17:45
  • $\begingroup$ In that case I don't think you should overfit. My work following Efron's showed that bootstrap 632 is better than leave-one-out version of cross-validation in estimating classification error rates for multivariate normal and other class conditional distributions that do not have heavy tails. But I do not know how k-fold cross-validation fares in that comparison. $\endgroup$ – Michael R. Chernick Sep 12 '12 at 18:33
2
$\begingroup$

I know it's an old thread, but anyway...

When I've designed algorithms to perform feature selection within a nested cross-validation, I've found that you can decrease overfitting to the inner segments by performing recursive elimination and making sure that you'll resample the inner segments for each iteration of the recursive feature elimination rather than performing full ranking/selection for each inner segment.

$\endgroup$
1
$\begingroup$

To answer my own question (I realize this material is probably actually stated elsewhere on this site):

I have found that when using complex classifier models (such as an SVM with an RBF kernel or an regularized discriminant classifier) there can be extremely large variance between folds in cross validation, if a large proportion of the data is withheld for testing within each fold.

I have found a practical strategy for nested cross-validation is to use leave one out cross validation for the outer loop and k-fold cross validation (in my case k=10) for the inner loop.

While this approach does mean that the final performance estimate has a larger variance, I have found it results in a 'better' model; as a larger proportion of data are used to find the optimal model in the inner CV loop.

$\endgroup$
  • 1
    $\begingroup$ Hi, would someone be kind enough to provide a reference describing the inner-loop/outer-loop nested cross-validation procedure? I've picked up bits and pieces and the high-level concept and rationale (only enough to be really dangerous ) by reading posts on stackexchange, but am still hazy on the details. I am not really expecting a stackexchange post will fully clarify things, but I'm open to any suggestion, of course. However, a journal article would probably be best. By the way, I am familiar with CV from sources like "Elmnts of Stat. Lrng." Thanks in advance for your help!! $\endgroup$ – clarpaul Mar 21 '17 at 23:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.