Does using Cross-Validation give you the green light to do exhaustive hyper-parameter searches?

Question

By hyper-parameters I mean not only the machine learning algorithm hyper-parameters (learning rate, etc.), but also hyper-parameters like "what's the ideal number of data points to use" or "which features should you include or not include" or even hyper-parameters associated with how the data is prepared.

My thought was that you can just run exhaustive searches (maybe even grid searches) on all kinds of hyper-parameter values and combinations, and just use cross-validation to avoid over-training, but I'm wondering whether or not that's contrary to best practices. I suppose that if you run enough tests on something as integral as the features used, you'll end up with a features that just happen to be highly correlated with your training/validation set, but then what's the guideline on "making sure you don't run too many tests"?

For reference this is with respect to a fast training model like xgboost (though this is more of a general question), and computational time isn't really an issue for me since I'm mostly only concerned with accuracy/performance, even if it takes me weeks to run the exhaustive searches. And if it matters I'm looking at a time-series dataset with datapoints in the order of 10,000s.

note: I apologize in advance since I'm just an amateur trying to do machine learning, any thoughts, even basic ones, would be highly appreciated!

Answer 1

No, using cross validation does not give the green light to run exhaustive searches over arbitrarily many hyperparameters. The usual goal of hyperparameter tuning (a.k.a. model selection) is to maximize generalization performance. Cross validation can be used for model selection because it provides a means to estimate generalization performance. However, because only finite data are available, it's possible for the model selection algorithm to overfit the validation set. That is, a particular choice of model or hyperparameters may happen to yield good performance on the validation set, but generalize poorly to unseen data from the underlying distribution. Overfitting the validation set may result in selecting models that either overfit or underfit the training data.

Overfitting during model selection is more likely when the validation set is smaller, or when searching over many models or hyperparameters. Obviously, this issue can be mitigated by increasing the amount of validation data (which includes using cross validation as opposed to simple holdout/split sample validation). Otherwise, if no further data are available, various mitigation strategies can be employed including regularization, early stopping, fully Bayesian approaches, and ensemble methods.

For more information, see:

Cawley, G. C., & Talbot, N. L. (2010). On over-fitting in model selection and subsequent selection bias in performance evaluation. The Journal of Machine Learning Research, 11, 2079-2107.