I'd like to select features, and evaluate model performance using nested cross validation. My question is that I have to split data in order to select features or not.

Additionally, is the following method appropriate? ( In the following method, feature selection was conducted without data split.)

  1. feature selection using correlation coefficient and Boruta algorithm
  2. Split the data into train and test at outer loop by 5-fold cross validation ( train-outer and test-outer)
  3. Split train-outer into train and valid at inner loop by 5-fold cross validation ( train-inner and valid-inner)
  4. Tune parameters of machine learning model (eg. support vector machine) by grid search using train-inner and valid-inner
  5. Train machine learning model using train-outer and evaluate model performance using test-outer

Feature selection should be performed after train-test split, so it should be part of your inner cv loop, treated like hyper-parameters of the model.

  • $\begingroup$ Thank you for your comment. In that case, different features would be selected at each outer loop. How should I decide which features are needed? $\endgroup$ – kifuc Oct 6 '20 at 13:57
  • $\begingroup$ if you apply cv for performance measurement, instead of a separate test set, this is always the problem. You won't have a final feature set or final hyper-parameter set $\endgroup$ – gunes Oct 6 '20 at 15:24
  • $\begingroup$ Thank you for your comment. My question was answered. $\endgroup$ – kifuc Oct 7 '20 at 13:55
  • $\begingroup$ Do you mean it's answered here or somewhere else? $\endgroup$ – gunes Oct 7 '20 at 13:57
  • 1
    $\begingroup$ I mean that you answered my question. $\endgroup$ – kifuc Oct 7 '20 at 16:04

I have released a package that can help implementing nested cross validation in Python (for the moment, it only works for binary classifiers). If you want to check it out, it's here:


It's my first Python package, so any comments, suggestions or critics will be more than welcome!!

I post it as an answer because nested cross validation is performed inside the main function and you don't have to take care of how to implement it.

Also, in the inner loop where the parameter and hyperparameter optimization takes place, you can add parameters of both model an post-processing pipeline (see the readme). This way, you can optimize the feature selection criteria -you could make it dependent of a parameter, like the number of features to select- in a safe way.

After the function is called, you get a model (that in fact will be a pipeline if there is a post-process to perform) and a complete report of what happened inside, so that you can assess the estimates of the different scoring functions. It comes with many options thay may be enough for a lot of common settings, I hope.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.