What it does

Bayesian optimization is a general, black box optimization strategy that works in the regime where the objective function may be stochastic and we don't necessarily have an expression for it, but we have the ability to evaluate it at any chosen point in parameter space. In the context of hyperparameter optimization, the parameters correspond to the hyperparameters of some learning algorithm, and the objective function is some measure of the learning algorithm's performance (typically on a validation set).

How it works

Bayesian optimization proceeds by sequentially choosing points at which to evaluate the objective function. It maintains a probabilistic model of the objective function, fit to the previously chosen points and corresponding function values. Current implementations commonly use Gaussian process regression or tree-based models. The probabilistic model is used together with an 'acquisition function' to choose the next point. The acquisition function strikes a balance between exploring new regions to reduce uncertainty about the objective function, and exploiting known structure to improve its value. After evaluating the objective function at the new point, the probabilistic model is updated, and the process is repeated.

Multiple acquisition functions have been proposed. For example, a popular strategy is to maximize the expected improvement over the currently-known best parameters. More complicated strategies are also possible. For example, we can maximize the expected improvement per second computing time, which also requires fitting a model to predict how long it will take to evaluate the objective function.

References

• Snoek et al. (2012). Practical Bayesian Optimization of Machine Learning Algorithms.

• Brochu et al. (2010). A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning.

• Eggensperger et al. (2013). Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters.

• Hutter et al. (2011). Sequential Model-Based Optimization for General Algorithm Configuration.

• Bergstra et al. (2011). Algorithms for Hyper-Parameter Optimization.

• Feurer et al. (2015). Efficient and Robust Automated Machine Learning.

• Big references list here (from the bayesopt documentation)

There are many existing implementations. Python libraries include spearmint, bayesopt, moe, auto-sklearn