0
$\begingroup$

I have a large dataset (1M+ samples) with 500 features. I need to create a predictive model that can be trained quickly. So, I want to perform an initial feature selection before building a classifier. I already know that most of the features are noisy (>70%) and irrelevant.

I am thinking of splitting the data into training (2/3) and test set (1/3). That is, I'll use the training set to build the classifier (with default parameter, no parameter tuning) and I'll evaluate its accuracy on the test set. Is this procedure biased/wrong?

  1. Run a feature selection algorithm (e.g. SVM-RFE) on the whole training set
  2. Select only the top 50 most important features
  3. Build a model (e.g. Random Forest) using only the features selected at (2.) with the whole training set
  4. Calculate the accuracy of the model over the test set

I assume that I don't need a cross-validation because at 1. and 2. there is no parameter to be tuned. In other words, I'm not looking to find the ideal number of features, I just want to know which are the top 50 and use them to build my predictive model.

Any thoughts or suggestions? Should I keep a part of the training set only to perform 1. and 2.? If so, why?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.