Which data for feature selection to get unbiased result? I have a 70 / 30 ratio for train / test data. I have a relatively small feature set (6 features), however, I still want to do feature selection to get rid of any redundant features (I'm guessing 1 of my features quite possibly is). Would running the feature selection on data later used for training / test bias the results? I know for cross-validation this is true, but how about a simple train / test set like mine?
 A: The short answer is yes. With your training/test split you have two basic choices for performing feature selection:


*

*Train your algorithm on the training set using a feature subset, and then test the performance on the training set.

*Train your algorithm on the training set using a feature subset, and then test the performance on the test set.


With 1 you can clearly suffer from overfitting as you're training and testing on the same dataset. 
With 2 you train and test on independent datasets, and therefore avoid overfitting in a way that approach 1 doesn't. However, the problem is that you evaluate your feature subsets by repeated testing against the test set. This means that you run the risk of one of your subsets happening to perform very well on the test set by chance (similar to the problem of multiple testing when determining p values).
I suppose ideally you would so something like take multiple samples of your training set (e.g. multiple random splits of it into training and test sets) to determine the best feature subset. After that you could train on your entire training set using your feature subset, and then test on your test set. Something like this means that your test set has never been seen before, and therefore the feature selection could not have specifically selected features for good performance on the test set. I found this paper (along with others that reference it) to be the most helpful when I was learning about this.
