Should be feature selection performed only on training data (or all data)? I went through some discussions and papers such as Guyon (2003) and Singhi and Liu (2006), but still not sure about right answer.
My experiment setup is as follows:
- Dataset: 50-healthy controls & 50-disease patients (cca 200 features that can be relevant to disease prediction).
- Task is to diagnose disease based on available features.
What I do is
- Take whole dataset and perform feature selection(FS). I keep only selected features for further processing
- Split to test and train, train classifier using train data and selected features. Then, apply classifier to test data (again using only selected features). Leave-one-out validation is used.
- obtain classification accuracy
- Averaging: repeat 1)-3) N times. $N=50$ (100).
I would agree that doing FS on whole dataset can introduce some bias, but my opinion is that it is "averaged out" during averaging (step 4). Is that correct? (Accuracy variance is $<2\%$)
1 Guyon, I. (2003) "An Introduction to Variable and Feature Selection", The Journal of Machine Learning Research, Vol. 3, pp. 1157-1182
2 Singhi, S.K. and Liu, H. (2006) "Feature Subset Selection Bias for Classification Learning", Proceeding ICML '06 Proceedings of the 23rd international conference on Machine learning, pp. 849-856