I have a classifier, and I am using leave one out cross-validation to assess its performance.

On each iteration, I divide the dataset into training and testing sets. The testing set is just the subject that I am going to evaluate (leave one out).

Now, I divide the training set into folds, and I do feature selection like this:

I run my filter feature selection algorithm on every fold. When I am done, I have a voting algorithm to obtain the final set with the features that were selected in each fold.

I understand that this procedure is adequate when you have a small sample like in my case (subjects = 30, features = 960).

My question is why, if at all, would it be a bad idea to do feature selection on the whole training set instead of dividing it into folds?

  • $\begingroup$ I may be wrong, but when number of features is much larger than the number of subjects, classification will not make any sense whatsoever... $\endgroup$
    – sashkello
    Jul 19, 2013 at 0:54
  • $\begingroup$ Yeah, you'll need some sort of penalty (elastic net or lasso) to estimate the model with more features than observations (see glmnet in R). $\endgroup$
    – wcampbell
    Jul 19, 2013 at 1:09
  • 1
    $\begingroup$ This appears to be a duplicate question, see my answer to a previous question here stats.stackexchange.com/questions/27750/… . Can you explain how your question differs from the earlier one? $\endgroup$ Jul 19, 2013 at 8:28
  • $\begingroup$ Answering to sashkello, you are right, I have more features than observations therefore I do feature selection. $\endgroup$
    – Diego
    Jul 19, 2013 at 13:45
  • $\begingroup$ Wcampbell: I am having good classification results using my feature selection algorithm. I have solved the model estimation part. My question is more about the rational to use cross validation inside the training set to do feature selection. Can I take the whole training set at once instead of partitioning it into folds? $\endgroup$
    – Diego
    Jul 19, 2013 at 13:48

1 Answer 1


The in-sample estimate of your prediction is generally overly optimistic. When you test your model on the same data on which it was trained, the model appears to fit very well. You get a more accurate estimate of your model's true predictive ability if you do cross validation - leave one out or k-fold. K-fold is much quicker than leave one out so that's what I use on larger samples.

By adding extra variables to the model, you always fit the sample data better. For example, in a linear regression, $R^{2}$ always increases when you add another variable to the model. That doesn't mean that adding that extra variable was a good idea. After adding too many variables, you end up over fitting the sample and don't do a good job of predicting new data. So, the main purpose of cross validation is to prevent over fitting the model and to ensure that you do the best possible job of prediction on an unseen data set.

  • $\begingroup$ I am not testing my classifier with the data I used for training. I am testing it with data it has been unseen by the classifier during the feature selection and training stages. That way I am avoiding overfitting. My question is about the implications of using the whole training set to select features instead of splitting the training set into folds and do feature selection on each fold. This is what I am doing right now and then I am voting for the best features on every fold. I need to clarify that my feature selection algorithm is a filter method. $\endgroup$
    – Diego
    Jul 19, 2013 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.