# Selecting multiple hyper-parameters via successive nested cross-validation

Selecting multiple hyper-parameters via successive nested cross-validation

I am currently working in a classification task on motion data. Each sample to classify is represented by a set of features computed using a sliding window of size X and with a step of length L. Once all the features are computed, I would like to do the following:

1. Currently, each sample is represented by 264 different features. The total number of samples depends of both X and L. From expert knowledge, I know that it might be possible to use a simpler model (with less features) for this classification task. I would like to prove it empirically, i.e, iteratively increase the number of features (up to 264) and see how a standard classifier behaves as the number of features increases.

2. Do some feature selection and keep as many features as indicated in the previous step.

3. Search the best hyper-parameters for all the models to be compared, e.g, SVM, Random Forest, etc...

4. Select the best model with the best parameters, based on its performance on the hold-out data set.

5. Train and deploy the final model.

I want to use classical K-fold cross-validation for each step. However, after reading all the related questions on stats (Model Tuning and Model Evaluation in Machine Learning, Feature selection and cross-validation, Model selection and cross-validation: The right way), I am still not sure about the best strategy to use. Intuitively and following the idea of nested cross-validation, I would do the following:

                    All-data
/      \
/        \
train_set   test_set (hold out) --> Final performance
/      \                            evaluation
/        \
train_set   test_set --> Hyper-parameter search
/      \
/        \
train_set   test_set --> Feature selection
/        \
/          \
train_set  test_set  --> Model complexity analysis (No. of features
/      \                        to use)
/        \
train_set   test_set --> Sliding window parameters


As the final model depends on the results of the previous steps and all of them are data-driven, I want to be sure that for each step, my decision is based on a not-biased test error and that there is no 'peeking into the future'. However, with so many nested cross-validation procedures, I am afraid of not having enough data for all them.

Do you think this is a good strategy? Should I use something else than cross-validation (bootstrapping for example)? Will this strategy lead to too optimistic scores? Once I find the best model, do I need to cross-validate the best hyper-parameters again on the whole dataset?

• It depends on how much data you have & how many instances of each class. The No. of features to use & Feature selection seem very similar to me: having decided to use features x1, x3, & x260, you have necessarily decided to use 3 features. Those steps could probably be combined. You may want to use a LASSO penalty to do both. – gung Oct 24 '15 at 15:13
• Hyperparameter and feature selection should be done jointly as they heavily affect each other, instead of doing one after the other. – Marc Claesen Oct 24 '15 at 16:29
• @gung In total I have 120 motion files (from 8 different subjects), 24 for each class. Thank for your suggestion about the features, I was thinking about it. It'll save me one split. However, I'm not really sure about when and how to select X and L parameters, because the other steps might highly depend on them. Maybe I should do the whole process for each possible X and L combination. – pcarreno Oct 27 '15 at 10:14
• @MarcClaesen, suppose I'm using a wrapper feature selection procedure as the one described here, do you mean to fit for example the C and gamma hyper-parameters of an SVM classifier at each step of my selection? Besides that, do you think the work-flow I propose is correct? – pcarreno Oct 27 '15 at 10:21