I have a large dataset (1M+ samples) with 500 features. I need to create a predictive model that can be trained quickly. So, I want to perform an initial feature selection before building a classifier. I already know that most of the features are noisy (>70%) and irrelevant.
I am thinking of splitting the data into training (2/3) and test set (1/3). That is, I'll use the training set to build the classifier (with default parameter, no parameter tuning) and I'll evaluate its accuracy on the test set. Is this procedure biased/wrong?
- Run a feature selection algorithm (e.g. SVM-RFE) on the whole training set
- Select only the top 50 most important features
- Build a model (e.g. Random Forest) using only the features selected at (2.) with the whole training set
- Calculate the accuracy of the model over the test set
I assume that I don't need a cross-validation because at 1. and 2. there is no parameter to be tuned. In other words, I'm not looking to find the ideal number of features, I just want to know which are the top 50 and use them to build my predictive model.
Any thoughts or suggestions? Should I keep a part of the training set only to perform 1. and 2.? If so, why?