So I need to select my hyperparameters and also my features. A full grid search of the space of hyperparameters and features is too computationally intensive, so what I am doing instead is for each fold of K-fold cross validation:

1) Tune hyperparameters using CV on the training set of the fold, using all features.

2) Select features using those hyperparameters from step 1.

3) Repeat for each fold

4) Final model is constructed on all the data using the N most prevalent features that were selected from each fold of CV. Hyperparameters will be tuned again using all the data in a CV loop.

Would there be a large downside from this method as compared to a full grid search? In essence I am doing a line search in each dimension of free parameters (finding the best value in 1 dimension, holding that constant then finding the best in the next dimension), rather than every single combination of parameter settings.

  • $\begingroup$ As this strategy will get stuck in local minima, I suggest that you try it several times with different initial parameter values. $\endgroup$
    – Bitwise
    May 2, 2014 at 18:20

1 Answer 1


The most important downside for searching along single parameters instead of optimizing them all together is that you ignore interactions. It is quite common that e.g. more than one parameter influences model complexity. In that case, you need to look at the interaction in order to sucessfully optimize the hyperparameters.

Depending on how large your data set is and how many models you compare, optimization strategies that return the maximum observed performance run into trouble (true for both grid search and your strategy). The reason is that searching through a large number of performance estimates for the maximum "skims" the variance of the performance estimate: you may just end up with a model and train/test split combination that accidentally happens to look good. Even worse, you may get several perfect looking combinations, and the optimization then cannot know which model to choose and thus becomes unstable.

  • $\begingroup$ What would be a better alternative approach than "optimization strategies that return the maximum observed performance"? $\endgroup$
    – Doopy
    Apr 6, 2021 at 9:56
  • $\begingroup$ @Doopy: using a heuristic that takes uncertainty into account. For a start, have a look at what the Elements of Statistical Learning call the one-standard-deviation rule $\endgroup$ Apr 6, 2021 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.