Why do you need a separate criterion (Sequential model-based global optimization) in hyperparameter tuning?

Question

In the paper 'Algorithms for Hyper-Parameter Optimization' (pdf), where they explain the 'Sequential Model-based Global optimization method (SMBO)', the authors made a comment that,

SMBO algorithms differ in what criterion they use to obtain the next candidate of hyper-parameters given a model (or surrogate of the expensive model $f$), and in model $f$ via observation history.

I unable to wrap my head around the need of criterion why not just use the best set of hyper-parameter that gives the best result for the surrogate of $f$. Elaborated, as author uses Gaussian processes as a surrogate - why you require criterion instead of querying for all candidate which is likely to improve just like in Gaussian regression.

Note: Here, SMBO refers to a method where hyper-parameters are being tuned for a given model $f$ when training operation is very expensive. In SMBO, a surrogate model of $f$ is used as it would be cheaper to train and thereby increase the hyper-parameter tuning operation.

Answer 1

I unable to wrap my head around the need of criterion why not just use the best set of hyper-parameter that gives the best result for the surrogate of $f$.

In GP surrogate models, the minimum of the surrogate is occurs at the argmin of the points queried so far, so choosing that point doesn't help at all -- you've already tested that hyper-parameter tuple, but when using SMBO, you're looking for promising points to try at the next iteration.

Elaborated, as author uses Gaussian processes as a surrogate - why you require criterion instead of querying for all candidate which is likely to improve just like in Gaussian regression.

The phrase "likely to improve" begs the question. Since the purpose of SMBO is to identify and test promising points, you'd need some way to compare the relative merits of two alternative points, which is what these criterions measure, using the previous points visited and the surrogate model to inform that measurement.