I'm using a supervised machine learning algorithm on some big data. There is much more features than observations. To reduce the number of features, I would like to do some feature selection. However, there is one thing I'm confused about. May I do the feature selection first, using data from all observations, and then do the training and testing of the classifier?
1
-
2$\begingroup$ no. feature selection=model buiding, and should only be done with training set. otherwise estimates provided by training set will be misleading. $\endgroup$– charlesCommented May 8, 2014 at 19:30
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
In any case you should always split your data into a development and a test set. You will look at the test set only once, which is when you evaluate the model you build, where model also includes the feature selection process.
Never use the test set during model selection.
Anything else is extremly dangerous in your setting. In case you need to look at the test set multiple times, you should have a range of test sets which you can add to your development set after each iteration.
