# Feature selection in the training set

I have a classifier, and I am using leave one out cross-validation to assess its performance.

On each iteration, I divide the dataset into training and testing sets. The testing set is just the subject that I am going to evaluate (leave one out).

Now, I divide the training set into folds, and I do feature selection like this:

I run my filter feature selection algorithm on every fold. When I am done, I have a voting algorithm to obtain the final set with the features that were selected in each fold.

I understand that this procedure is adequate when you have a small sample like in my case (subjects = 30, features = 960).

My question is why, if at all, would it be a bad idea to do feature selection on the whole training set instead of dividing it into folds?

• I may be wrong, but when number of features is much larger than the number of subjects, classification will not make any sense whatsoever... – sashkello Jul 19 '13 at 0:54
• Yeah, you'll need some sort of penalty (elastic net or lasso) to estimate the model with more features than observations (see glmnet in R). – wcampbell Jul 19 '13 at 1:09
• This appears to be a duplicate question, see my answer to a previous question here stats.stackexchange.com/questions/27750/… . Can you explain how your question differs from the earlier one? – Dikran Marsupial Jul 19 '13 at 8:28
• Answering to sashkello, you are right, I have more features than observations therefore I do feature selection. – Diego Jul 19 '13 at 13:45
• Wcampbell: I am having good classification results using my feature selection algorithm. I have solved the model estimation part. My question is more about the rational to use cross validation inside the training set to do feature selection. Can I take the whole training set at once instead of partitioning it into folds? – Diego Jul 19 '13 at 13:48

By adding extra variables to the model, you always fit the sample data better. For example, in a linear regression, $R^{2}$ always increases when you add another variable to the model. That doesn't mean that adding that extra variable was a good idea. After adding too many variables, you end up over fitting the sample and don't do a good job of predicting new data. So, the main purpose of cross validation is to prevent over fitting the model and to ensure that you do the best possible job of prediction on an unseen data set.