When and why use Bayesian optimization, instead of gradient descent?
Which one is better for which cases?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
I'm assuming that by Bayesian optimization you mean the standard method of fitting a Gaussian process or similar model to your observations, defining an acquisition function such as expected improvement or upper confidence bound, and querying the function at the maximum of that acquisition function.
The most immediate difference is that Bayesian optimization is applicable when you don't know the gradients. If you can cheaply compute gradients of your function, you'll want to use a method that can incorporate those, since they can be extremely helpful in understanding the function. If you can't easily compute gradients and need to resort to finite differences approximations, in most cases you really don't want to do that.
BO assumes that the function is fairly smooth (as defined by your kernel in the GP) but not convex. Gradient descent, if you want to find a global maximum, assumes convexity as well as some degree of smoothness (as used in the step size parameter).
BO tries to minimize the number of calls to the objective function. If it is expensive to calculate, e.g. because it requires a lot of computation or even some interaction with the outside world, this is highly desirable. If the objective is cheap to calculate, it may be faster to not bother with the amount of side computation needed.
BO typically does not scale well to high-dimensional functions, either statistically or computationally. Recent work has begun to address this issue, from various tacks. Gradient descent and similar methods often scale reasonably to higher dimensions, if you're careful.
BO also doesn't typically scale well to evaluating the function many times, since GP inference is cubic in the number of input points. There's been a lot of recent work on speeding up GPs or using similar more scalable methods, which may or may not address this problem for your needs.