I have a large dataset (1M+ samples) with 500 features. I need to create a predictive model that can be trained quickly. So, I want to perform an initial feature selection before building a classifier. I already know that most of the features are noisy (>70%) and irrelevant.

I am thinking of splitting the data into training (2/3) and test set (1/3). That is, I'll use the training set to build the classifier (with default parameter, no parameter tuning) and I'll evaluate its accuracy on the test set. Is this procedure biased/wrong?

  1. Run a feature selection algorithm (e.g. SVM-RFE) on the whole training set
  2. Select only the top 50 most important features
  3. Build a model (e.g. Random Forest) using only the features selected at (2.) with the whole training set
  4. Calculate the accuracy of the model over the test set

I assume that I don't need a cross-validation because at 1. and 2. there is no parameter to be tuned. In other words, I'm not looking to find the ideal number of features, I just want to know which are the top 50 and use them to build my predictive model.

Any thoughts or suggestions? Should I keep a part of the training set only to perform 1. and 2.? If so, why?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.