2
$\begingroup$

I'd like to select features, and evaluate model performance using nested cross validation. My question is that I have to split data in order to select features or not.

Additionally, is the following method appropriate? ( In the following method, feature selection was conducted without data split.)

  1. feature selection using correlation coefficient and Boruta algorithm
  2. Split the data into train and test at outer loop by 5-fold cross validation ( train-outer and test-outer)
  3. Split train-outer into train and valid at inner loop by 5-fold cross validation ( train-inner and valid-inner)
  4. Tune parameters of machine learning model (eg. support vector machine) by grid search using train-inner and valid-inner
  5. Train machine learning model using train-outer and evaluate model performance using test-outer
Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
1
$\begingroup$

Feature selection should be performed after train-test split, so it should be part of your inner cv loop, treated like hyper-parameters of the model.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for your comment. In that case, different features would be selected at each outer loop. How should I decide which features are needed? $\endgroup$
    – kifuc
    Oct 6, 2020 at 13:57
  • $\begingroup$ if you apply cv for performance measurement, instead of a separate test set, this is always the problem. You won't have a final feature set or final hyper-parameter set $\endgroup$
    – gunes
    Oct 6, 2020 at 15:24
  • $\begingroup$ Thank you for your comment. My question was answered. $\endgroup$
    – kifuc
    Oct 7, 2020 at 13:55
  • $\begingroup$ Do you mean it's answered here or somewhere else? $\endgroup$
    – gunes
    Oct 7, 2020 at 13:57
  • 1
    $\begingroup$ I mean that you answered my question. $\endgroup$
    – kifuc
    Oct 7, 2020 at 16:04
0
$\begingroup$

I have released a package that can help implementing nested cross validation in Python (for the moment, it only works for binary classifiers). If you want to check it out, it's here:

https://github.com/JaimeArboleda/nestedcvtraining

It's my first Python package, so any comments, suggestions or critics will be more than welcome!!

I post it as an answer because nested cross validation is performed inside the main function and you don't have to take care of how to implement it.

Also, in the inner loop where the parameter and hyperparameter optimization takes place, you can add parameters of both model an post-processing pipeline (see the readme). This way, you can optimize the feature selection criteria -you could make it dependent of a parameter, like the number of features to select- in a safe way.

After the function is called, you get a model (that in fact will be a pipeline if there is a post-process to perform) and a complete report of what happened inside, so that you can assess the estimates of the different scoring functions. It comes with many options thay may be enough for a lot of common settings, I hope.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.