Feature selection: nested cross validation I'd like to select features, and evaluate model performance using nested cross validation.
My question is that I have to split data in order to select features or not.
Additionally, is the following method appropriate?
( In the following method, feature selection was conducted without data split.)

*

*feature selection using correlation coefficient and Boruta algorithm

*Split the data into train and test at outer loop by 5-fold cross validation ( train-outer and test-outer)

*Split train-outer into train and valid at inner loop by 5-fold cross validation ( train-inner and valid-inner)

*Tune parameters of machine learning model (eg. support vector machine) by grid search using train-inner and valid-inner

*Train machine learning model using train-outer and evaluate model performance using test-outer

 A: Feature selection should be performed after train-test split, so it should be part of your inner cv loop, treated like hyper-parameters of the model.
A: I have released a package that can help implementing nested cross validation in Python (for the moment, it only works for binary classifiers). If you want to check it out, it's here:
https://github.com/JaimeArboleda/nestedcvtraining
It's my first Python package, so any comments, suggestions or critics will be more than welcome!!
I post it as an answer because nested cross validation is performed inside the main function and you don't have to take care of how to implement it.
Also, in the inner loop where the parameter and hyperparameter optimization takes place, you can add parameters of both model an post-processing pipeline (see the readme). This way, you can optimize the feature selection criteria -you could make it dependent of a parameter, like the number of features to select- in a safe way.
After the function is called, you get a model (that in fact will be a pipeline if there is a post-process to perform) and a complete report of what happened inside, so that you can assess the estimates of the different scoring functions. It comes with many options thay may be enough for a lot of common settings, I hope.
