I'd like to select features, and evaluate model performance using nested cross validation. My question is that I have to split data in order to select features or not.

Additionally, is the following method appropriate? ( In the following method, feature selection was conducted without data split.)

  1. feature selection using correlation coefficient and Boruta algorithm
  2. Split the data into train and test at outer loop by 5-fold cross validation ( train-outer and test-outer)
  3. Split train-outer into train and valid at inner loop by 5-fold cross validation ( train-inner and valid-inner)
  4. Tune parameters of machine learning model (eg. support vector machine) by grid search using train-inner and valid-inner
  5. Train machine learning model using train-outer and evaluate model performance using test-outer

Feature selection should be performed after train-test split, so it should be part of your inner cv loop, treated like hyper-parameters of the model.

  • $\begingroup$ Thank you for your comment. In that case, different features would be selected at each outer loop. How should I decide which features are needed? $\endgroup$ – kifuc Oct 6 '20 at 13:57
  • $\begingroup$ if you apply cv for performance measurement, instead of a separate test set, this is always the problem. You won't have a final feature set or final hyper-parameter set $\endgroup$ – gunes Oct 6 '20 at 15:24
  • $\begingroup$ Thank you for your comment. My question was answered. $\endgroup$ – kifuc Oct 7 '20 at 13:55
  • $\begingroup$ Do you mean it's answered here or somewhere else? $\endgroup$ – gunes Oct 7 '20 at 13:57
  • 1
    $\begingroup$ I mean that you answered my question. $\endgroup$ – kifuc Oct 7 '20 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.