Selecting multiple hyper-parameters via successive nested cross-validation

Question

Selecting multiple hyper-parameters via successive nested cross-validation

I am currently working in a classification task on motion data. Each sample to classify is represented by a set of features computed using a sliding window of size X and with a step of length L. Once all the features are computed, I would like to do the following:

1. Currently, each sample is represented by 264 different features. The total number of samples depends of both X and L. From expert knowledge, I know that it might be possible to use a simpler model (with less features) for this classification task. I would like to prove it empirically, i.e, iteratively increase the number of features (up to 264) and see how a standard classifier behaves as the number of features increases.

2. Do some feature selection and keep as many features as indicated in the previous step.

3. Search the best hyper-parameters for all the models to be compared, e.g, SVM, Random Forest, etc...

4. Select the best model with the best parameters, based on its performance on the hold-out data set.

5. Train and deploy the final model.

I want to use classical K-fold cross-validation for each step. However, after reading all the related questions on stats (Model Tuning and Model Evaluation in Machine Learning, Feature selection and cross-validation, Model selection and cross-validation: The right way), I am still not sure about the best strategy to use. Intuitively and following the idea of nested cross-validation, I would do the following:

                    All-data 
                    /      \
                   /        \
                train_set   test_set (hold out) --> Final performance  
                 /      \                            evaluation
                /        \
              train_set   test_set --> Hyper-parameter search 
             /      \               
            /        \  
         train_set   test_set --> Feature selection
        /        \
       /          \
   train_set  test_set  --> Model complexity analysis (No. of features  
   /      \                        to use)
  /        \
 train_set   test_set --> Sliding window parameters

As the final model depends on the results of the previous steps and all of them are data-driven, I want to be sure that for each step, my decision is based on a not-biased test error and that there is no 'peeking into the future'. However, with so many nested cross-validation procedures, I am afraid of not having enough data for all them.

Do you think this is a good strategy? Should I use something else than cross-validation (bootstrapping for example)? Will this strategy lead to too optimistic scores? Once I find the best model, do I need to cross-validate the best hyper-parameters again on the whole dataset?

Answer 1

I have done things like that in the past, where I needed to optimize both hyperparameters of the model and parameters of the post-processing pipeline (I include feature selection there). But it's very expensive computationally speaking.

So finally I tried to make my own standard solution and I have released a package that can help implementing nested cross validation in Python (for the moment, it only works for binary classifiers). If you want to check it out, it's here:

https://github.com/JaimeArboleda/nestedcvtraining

It's my first Python package, so any comments, suggestions or critics will be more than welcome!!

I post it as an answer because nested cross validation is performed inside the main function and you don't have to take care of how to implement it. Anyway, the code is visible in my github account, so that you can check how I implemented it if you are curious. Also, the readme explain it in detail (although, maybe, not very clearly!).

After the function is called, you get a model (that in fact will be a pipeline if there is a post-process to perform) and a complete report of what happened inside, so that you can assess the estimates of the different scoring functions. It comes with many options thay may be enough for a lot of common settings, I hope.

If you have any doubt, comment, whatever, you can contact me through github.

Answer 2

The important thing is that the entire model fitting procedure is performed independently in each fold, including all of the feature and hyper-parameter tuning, as that is what is needed to avoid bias due to over-fitting in model selection.

I implement this by making a function, something like:

model = train_model(x, t)

where x is the input and t is the targets for the training data, and another function:

y = generate_outputs(model, x)

where x is the test inputs and y will be the test predictions from the model.

Now train_model will encode how the hyper-parameters etc. are tuned. As @marc_claesen suggests, it is better to perform hyper-paremeter and feature selection jointly, especially for things like SVM* where the regularisation parameter depends on the feature set. I then use those two functions for implementing the outer cross-validation.

I would probably perform all of the other steps jointly using a single inner-cross-validation. Note however the more you tune the model using cross-validation, the more you will tend to overfit the model, so exhaustively searching for the best possible model tends to end up giving a rather poor one. In many machine learning challenges (e.g. at conferences), the models at the top of the leader board tend to be towards the bottom of the final rankings because they have often been over-fitted to the leaderboard data by too much manual model selection (I call this "cyberML", which is a bit like "autoML" but where the operator has become part of the machine).

*for SVMs I normally find feature selection make generalisation worse rather than better, at least for linear SVMs, as they are based on a learning bound that is independent of the size of the feature space, so regularisation tends to work well. The generalisation theory for feature selection is much less rigorous, and invalidates the generalisation bounds for the SVM, and tuning one continuous regularisation parameter tends to be less prone to over-fitting the model selection criterion than fitting one binary hyper-parameter for each feature (which is essentially what feature selection is doing). So for things like SVMs, I tend only to do feature selection if selecting the features is a primary goal for the analysis.

I agree with @gung's suggestion of using e.g. LASSO penalty for feature selection (fewer degrees of freedom, and less pressure on CV to find the best model)