Selecting multiple hyper-parameters via successive nested cross-validation

I am currently working in a classification task on motion data. Each sample to classify is represented by a set of features computed using a sliding window of size X and with a step of length L. Once all the features are computed, I would like to do the following:

  1. Currently, each sample is represented by 264 different features. The total number of samples depends of both X and L. From expert knowledge, I know that it might be possible to use a simpler model (with less features) for this classification task. I would like to prove it empirically, i.e, iteratively increase the number of features (up to 264) and see how a standard classifier behaves as the number of features increases.

  2. Do some feature selection and keep as many features as indicated in the previous step.

  3. Search the best hyper-parameters for all the models to be compared, e.g, SVM, Random Forest, etc...

  4. Select the best model with the best parameters, based on its performance on the hold-out data set.

  5. Train and deploy the final model.

I want to use classical K-fold cross-validation for each step. However, after reading all the related questions on stats (Model Tuning and Model Evaluation in Machine Learning, Feature selection and cross-validation, Model selection and cross-validation: The right way), I am still not sure about the best strategy to use. Intuitively and following the idea of nested cross-validation, I would do the following:

                    /      \
                   /        \
                train_set   test_set (hold out) --> Final performance  
                 /      \                            evaluation
                /        \
              train_set   test_set --> Hyper-parameter search 
             /      \               
            /        \  
         train_set   test_set --> Feature selection
        /        \
       /          \
   train_set  test_set  --> Model complexity analysis (No. of features  
   /      \                        to use)
  /        \
 train_set   test_set --> Sliding window parameters

As the final model depends on the results of the previous steps and all of them are data-driven, I want to be sure that for each step, my decision is based on a not-biased test error and that there is no 'peeking into the future'. However, with so many nested cross-validation procedures, I am afraid of not having enough data for all them.

Do you think this is a good strategy? Should I use something else than cross-validation (bootstrapping for example)? Will this strategy lead to too optimistic scores? Once I find the best model, do I need to cross-validate the best hyper-parameters again on the whole dataset?

  • 2
    $\begingroup$ It depends on how much data you have & how many instances of each class. The No. of features to use & Feature selection seem very similar to me: having decided to use features x1, x3, & x260, you have necessarily decided to use 3 features. Those steps could probably be combined. You may want to use a LASSO penalty to do both. $\endgroup$ – gung Oct 24 '15 at 15:13
  • 2
    $\begingroup$ Hyperparameter and feature selection should be done jointly as they heavily affect each other, instead of doing one after the other. $\endgroup$ – Marc Claesen Oct 24 '15 at 16:29
  • $\begingroup$ @gung In total I have 120 motion files (from 8 different subjects), 24 for each class. Thank for your suggestion about the features, I was thinking about it. It'll save me one split. However, I'm not really sure about when and how to select X and L parameters, because the other steps might highly depend on them. Maybe I should do the whole process for each possible X and L combination. $\endgroup$ – pcarreno Oct 27 '15 at 10:14
  • 1
    $\begingroup$ @MarcClaesen, suppose I'm using a wrapper feature selection procedure as the one described here, do you mean to fit for example the C and gamma hyper-parameters of an SVM classifier at each step of my selection? Besides that, do you think the work-flow I propose is correct? $\endgroup$ – pcarreno Oct 27 '15 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.