5
$\begingroup$

In the paper 'Algorithms for Hyper-Parameter Optimization' (pdf), where they explain the 'Sequential Model-based Global optimization method (SMBO)', the authors made a comment that,

SMBO algorithms differ in what criterion they use to obtain the next candidate of hyper-parameters given a model (or surrogate of the expensive model $f$), and in model $f$ via observation history.

I unable to wrap my head around the need of criterion why not just use the best set of hyper-parameter that gives the best result for the surrogate of $f$. Elaborated, as author uses Gaussian processes as a surrogate - why you require criterion instead of querying for all candidate which is likely to improve just like in Gaussian regression.

Note: Here, SMBO refers to a method where hyper-parameters are being tuned for a given model $f$ when training operation is very expensive. In SMBO, a surrogate model of $f$ is used as it would be cheaper to train and thereby increase the hyper-parameter tuning operation.

| cite | improve this question | | | | |
$\endgroup$
3
$\begingroup$

I unable to wrap my head around the need of criterion why not just use the best set of hyper-parameter that gives the best result for the surrogate of $f$.

In GP surrogate models, the minimum of the surrogate is occurs at the argmin of the points queried so far, so choosing that point doesn't help at all -- you've already tested that hyper-parameter tuple, but when using SMBO, you're looking for promising points to try at the next iteration.

Elaborated, as author uses Gaussian processes as a surrogate - why you require criterion instead of querying for all candidate which is likely to improve just like in Gaussian regression.

The phrase "likely to improve" begs the question. Since the purpose of SMBO is to identify and test promising points, you'd need some way to compare the relative merits of two alternative points, which is what these criterions measure, using the previous points visited and the surrogate model to inform that measurement.

| cite | improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.