12
$\begingroup$

I am fairly new to machine learning and statistics but I was wondering why bayesian optimization is not referred more often online when learning machine learning to optimize your algorithm hyperparameters? For example using a framework like this one: https://github.com/fmfn/BayesianOptimization

Does bayesian optimization of your hyperparameters have any limitation or major disadvantage over techniques like grid search or random search?

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ No free lunch in search and optimization. In general, unless cost function evaluation is rather costly and the dimensionality of the problem somewhat small, BO is usually not the answer. The field of Mathematical Optimisation did not become obsolete because of the discovery of Gaussian Processes. $\endgroup$
    – usεr11852
    Commented Aug 10, 2017 at 20:57
  • 3
    $\begingroup$ Great answers. But why do you post them in comments? $\endgroup$
    – Jan Kukacka
    Commented Feb 2, 2018 at 10:23
  • $\begingroup$ @JanKukacka Good point. I've moved my comments to an answer. $\endgroup$
    – Sycorax ♦
    Commented Jun 18, 2018 at 3:12

1 Answer 1

Reset to default
14
$\begingroup$
  1. results are sensitive to parameters of the surrogate model, which are typically fixed at some value; this underestimates uncertainty; or else you have to be fully Bayesian and marginalize over hyper parameter distributions, which can be expensive and unwieldy.
  2. it takes a dozen or so samples to get a good surrogate surface in 2 or 3 dimensions of search space; increasing dimensionality of the search space requires yet more samples
  3. Bayesian optimization itself depends on an optimizer to search the surrogate surface, which has its own costs -- this problem is (hopefully) cheaper to evaluate than the original problem, but it is still a non-convex box-constrained optimization problem (i.e., difficult!)
  4. estimating the BO model itself has costs

To state it another way, BO is an attempt to keep the number of function evaluations to a minimum, and get the most "bang for the buck" from each evaluation. This is important if you're conducting destructive tests, or just doing a simulation that takes an obscene amount of time to execute. But in all but the most expensive cases, apply pure random search and call it a day! (Or LIPO if your problem is amenable to its assumptions.) It can save you a number of headaches, such as optimizing your Bayesian Optimization program.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.