Selecting multiple hyper-parameters via successive nested cross-validation
I am currently working in a classification task on motion data. Each sample to classify is represented by a set of features computed using a sliding window of size X and with a step of length L. Once all the features are computed, I would like to do the following:
Currently, each sample is represented by 264 different features. The total number of samples depends of both X and L. From expert knowledge, I know that it might be possible to use a simpler model (with less features) for this classification task. I would like to prove it empirically, i.e, iteratively increase the number of features (up to 264) and see how a standard classifier behaves as the number of features increases.
Do some feature selection and keep as many features as indicated in the previous step.
Search the best hyper-parameters for all the models to be compared, e.g, SVM, Random Forest, etc...
Select the best model with the best parameters, based on its performance on the hold-out data set.
Train and deploy the final model.
I want to use classical K-fold cross-validation for each step. However, after reading all the related questions on stats (Model Tuning and Model Evaluation in Machine Learning, Feature selection and cross-validation, Model selection and cross-validation: The right way), I am still not sure about the best strategy to use. Intuitively and following the idea of nested cross-validation, I would do the following:
All-data / \ / \ train_set test_set (hold out) --> Final performance / \ evaluation / \ train_set test_set --> Hyper-parameter search / \ / \ train_set test_set --> Feature selection / \ / \ train_set test_set --> Model complexity analysis (No. of features / \ to use) / \ train_set test_set --> Sliding window parameters
As the final model depends on the results of the previous steps and all of them are data-driven, I want to be sure that for each step, my decision is based on a not-biased test error and that there is no 'peeking into the future'. However, with so many nested cross-validation procedures, I am afraid of not having enough data for all them.
Do you think this is a good strategy? Should I use something else than cross-validation (bootstrapping for example)? Will this strategy lead to too optimistic scores? Once I find the best model, do I need to cross-validate the best hyper-parameters again on the whole dataset?