I would like to implement Bayesian hyperparameter tuning in Python, but I am struggling to find a complete explanation of how it is exactly working. As anyone links, books or papers I could read about it with detailed explanation? Would really appreciate it. Thanks, Umberto

  • $\begingroup$ Closely related: stats.stackexchange.com/questions/193306/… $\endgroup$ – Sycorax Jul 30 '18 at 16:25
  • $\begingroup$ If it is still of interest for someone stumbling on this issue, I had the same question and summarized what I understood in a post about hyperparameters tuning, mainly about grid search vs random search vs bayesian optimization. It is less technical than the great papers in user20160's answer so it might be an intro to this issue. $\endgroup$ – JPro Nov 20 '19 at 17:23
  • $\begingroup$ This tutorial for Bayesian optimization is for practical use case. It is written in R. bearloga.github.io/bayesopt-tutorial-r $\endgroup$ – nan hu Nov 20 '19 at 17:49

What it does

Bayesian optimization is a general, black box optimization strategy that works in the regime where the objective function may be stochastic and we don't necessarily have an expression for it, but we have the ability to evaluate it at any chosen point in parameter space. In the context of hyperparameter optimization, the parameters correspond to the hyperparameters of some learning algorithm, and the objective function is some measure of the learning algorithm's performance (typically on a validation set).

How it works

Bayesian optimization proceeds by sequentially choosing points at which to evaluate the objective function. It maintains a probabilistic model of the objective function, fit to the previously chosen points and corresponding function values. Current implementations commonly use Gaussian process regression or tree-based models. The probabilistic model is used together with an 'acquisition function' to choose the next point. The acquisition function strikes a balance between exploring new regions to reduce uncertainty about the objective function, and exploiting known structure to improve its value. After evaluating the objective function at the new point, the probabilistic model is updated, and the process is repeated.

Multiple acquisition functions have been proposed. For example, a popular strategy is to maximize the expected improvement over the currently-known best parameters. More complicated strategies are also possible. For example, we can maximize the expected improvement per second computing time, which also requires fitting a model to predict how long it will take to evaluate the objective function.


  • Snoek et al. (2012). Practical Bayesian Optimization of Machine Learning Algorithms.

  • Brochu et al. (2010). A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning.

  • Eggensperger et al. (2013). Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters.

  • Hutter et al. (2011). Sequential Model-Based Optimization for General Algorithm Configuration.

  • Bergstra et al. (2011). Algorithms for Hyper-Parameter Optimization.

  • Feurer et al. (2015). Efficient and Robust Automated Machine Learning.

  • Big references list here (from the bayesopt documentation)

There are many existing implementations. Python libraries include spearmint, bayesopt, moe, auto-sklearn

| cite | improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.