@DennisSoemers has a great solution. I'll add a two similar solutions that are a bit more explicit and based on Feature Engineering and Selection: A Practical Approach for Predictive Models by Max Kuhn and Kjell Johnson.
Kuhn uses the term
resample to describe a
fold of a dataset, but the dominant term on StackExchange seems to be
fold, so I will use the term
Option 1 - nested search
If compute power is not a limiting factor, a nested validation approach is recommended, in which there are 3 levels of nesting:
1) the external folds, each fold with a different feature subset
2) the internal folds, each fold with a hyperparameter search
3) the internal folds of each hyperparameter search, each fold with a different hyperparameter set.
Here's the algorithm:
-> Split data into train and test sets.
-> For each external fold of train set:
-> Select feature subset.
-> Split into external train and test sets.
-> For each internal fold of external train set:
-> Split into internal train and test sets.
-> Perform hyperparameter tuning on the internal train set. Note that this
step is another level of nesting in which the internal train set is split
into multiple folds and different hyperparameter sets are trained and tested on
-> Examine the performance of the best hyperparameter tuned model
from each of the inner test folds. If performance is consistent, redo
the internal hyperparameter tuning step on the entire external train set.
-> Test the model with the best hyperparameter set on the external test set.
-> Choose the feature set with the best external test score.
-> Retrain the model on all of the training data using the best feature set
and best hyperparameters for that feature set.
Image from Chapter 11.2: Simple Filters
-> Select feature subset step is implied to be random, but there are other techniques, which are outlined in the book in Chapter 11.
To clarify the
-> Perform hyperparameter tuning step, you can read about the recommended approach of nested cross validation. The idea is to test the robustness of a training process by repeatedly performing the training and testing process on different folds of the data, and looking at the average of test results.
Option 2 - separate hyperparameter and feature selection search
-> Split data into hyperameter_train, feature_selection_train, and test sets.
-> Select a reasonable subset of features using expert knowledge.
-> Perform nested cross validation with the initial features and the
hyperparameter_train set to find the best hyperparameters as outlined in option 1.
-> Use the best hyperparameters and the feature_selection_train set to find
the best set of features. Again, this process could be nested cross
validation or not, depending on the computational cost that it would take
and the cost that is tolerable.
Here's how Kuhn and Johsnon phrase the process:
When combining a global search method with a model that has tuning
parameters, we recommend that, when possible, the feature set first be
winnowed down using expert knowledge about the problem. Next, it is
important to identify a reasonable range of tuning parameter values.
If a sufficient number of samples are available, a proportion of them
can be split off and used to find a range of potentially good
parameter values using all of the features. The tuning parameter
values may not be the perfect choice for feature subsets, but they
should be reasonably effective for finding an optimal subset.
Chapter 12.5: Global Search Methods