0
$\begingroup$

I was reading up on Bayesian Optimization and in one of the articles, I came across the following passage.

We could just use the surrogate score directly. Alternately, given that we have chosen a Gaussian Process model as the surrogate function, we can use the probabilistic information from this model in the acquisition function to calculate the probability that a given sample is worth evaluating.

I am trying the understand why this would be necessary? Wouldn't simply using the scores suffice? If we are looking to maximize an objective function and find a sample that gives the highest value on the surrogate model, what's the harm in picking that sample directly?

Also, what's the advantage of using acquisition functions based on the probability of improvement, expected improvement, etc?

I am trying to find a justification for the extra step.

$\endgroup$
0
$\begingroup$

The point of using a surrogate model is that it lets you make an educated guess about what point to evaluate. Imagine that you are optimizing hyperparameters of a model that takes 24h to train, in such case, with large grid of hyperparameters, it could take years to train and evaluate the model for all the combinations of hyperparameters. So looking at the "actual scores" wouldn't really work in here. In Bayesian optimization, we are using the surrogate model to make guesstimate on what could be the score for a combination of parameters, and then we are verifying those guesses on the most interesting candidates, instead all of the possible candidates.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.