I know that you should separate your data into training and validation sets before doing feature selection, to avoid getting too good, false results on cross validation.

But, I have seen people say that you should also avoid doing feature selection on the same data set that you train your model on, as to avoid overfitting on that data.

What some suggest is that you split your training data into 3 sets, a training, validation and testing dataset. You train your model on the training dataset, do feature selection on the validation set, and evaluate your model on the testing dataset.

Is this overfitting really something to worry about, and if yes, is the above method a good way to deal with it?

  • 1
    $\begingroup$ Feature selection is generally performed before training a model, not after, in order to eliminate noisy or uninformative features that can confound the algorithm. Some classification methods are implicitly feature-selective, and for those it's not possible to separate the feature selection and model-building steps. $\endgroup$ – Nuclear Wang Jul 16 '18 at 12:49
  • $\begingroup$ maybe what you mean is model selection? Then everything makes sense. stats.stackexchange.com/questions/19048/… $\endgroup$ – Xiaoxiong Lin Jul 16 '18 at 13:49
  • $\begingroup$ @XiaoxiongLin No, I meant feature selection. I really don't know how could I explain the question any further. $\endgroup$ – Ian Dzindo Jul 16 '18 at 17:00
  • $\begingroup$ Can you specify an example of feature selection? I'm not an expert, but if you mean e.g. determining lambda for LASSO is a way of feature selection, it could lead to overfitting. $\endgroup$ – Xiaoxiong Lin Jul 17 '18 at 13:38
  • $\begingroup$ @XiaoxiongLin Feature selection as in SelectKBest or SelectPercentile. $\endgroup$ – Ian Dzindo Jul 18 '18 at 6:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.