I want to use Bayesian optimization to search a space of hyperparameters for a neural network model. My objective function for this optimization is validation set accuracy.

In addition, I want to perform cross-validation such that I can get a good estimate of the best hyperparameters for test-set performance when training on the whole training set.

Given these two desires, and a search space for the bayesian optimization procedure, I can see two options for how to conduct the experiment at a high level.

In the first, I split the training set into N folds. Then for each fold I run the entire Bayesian optimization process, this produces N sets of values for my hyperparameters, a best set for each fold. I choose the best set among those from the N folds and retrain on the whole training set. This is cross-validation in the classical setting.

In the second, within each evaluation of the objective function for the bayesian optimization, I perform cross-validation to find the best validation set accuracy. Thus I train the model with the fixed hyperparameters that are the point in the search space being evaluated. I do this for each training set fold, and evaluate on each respective validation fold. Then, the objective function value returned for this evaluation in the Bayesian optimization procedure is the best validation set accuracy.

My question is this: are these two approaches equivalent? Is the latter a statistically valid estimate of the best parameters, or something else entirely? Are there any (dis)advantages either way? The latter is considerably easier to implement given the Bayesian optimization framework I'm using (Optuna).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.