13
$\begingroup$

So, we know that random search works better than grid search, but a more recent approach is Bayesian optimization (using gaussian processes). I've looked up a comparison between the two, and found nothing. I know that at Stanford's cs231n they mention only random search, but it is possible that they wanted to keep things simple.

My question is: which approach is generally better, and if the answer is "sometimes random search, sometimes Bayesian" when should I prefer one method over another?

$\endgroup$
  • 2
    $\begingroup$ Google is selling their deep learning cloud services now and pushing a feature that automatically tunes your hyperparameters with Bayesian optimization...of course claiming it does the best and is faster as well (searching the hyperspace more efficiently). There are several papers out there that evaluate BO vs RS as well showing BO doing just slightly better. IMO from what I've seen, the diff is something you'd care about more in a Kaggle competition than real life. $\endgroup$ – JPJ Dec 7 '17 at 5:52
8
$\begingroup$

I think that the answer here is the same as everywhere in data science: it depends on the data :-)

It might happen that one method outperforms another (here https://arimo.com/data-science/2016/bayesian-optimization-hyperparameter-tuning/ people compare Bayesian hyperparameter optimization and achieve a better result on the San Francisco crime kaggle challenge than with random search), however I doubt that there is a general rule for that. You can see a nice gif here (http://blog.revolutionanalytics.com/2016/06/bayesian-optimization-of-machine-learning-models.html) where people show the 'path' that Bayesian optimization takes in the landscape of hyperparameters, in particular, it does not seem as if it outperforms random search in general...

I think the reason why people tend to use Bayesian hyperparameter optimization is that it just takes less training steps in order to achieve a comparable result as compared to random search with a sufficiently high number of experiments.

Summarising in one sentence:

*When training time is critical, use Bayesian hyperparameter optimization and if time is not an issue, select one of both... *

Usually I am too lazy to implement the Bayesian stuff with Gaussian Processes if I can achieve the same result with random search... I just train Gradient Bossting ensembles on 'few' data, so for me, time is not an issue...

$\endgroup$
5
$\begingroup$

Bayesian optimization is better, because it makes smarter decisions. You can check this article in order to learn more: Hyperparameter optimization for neural networks. This articles also has info about pros and cons for both methods + some extra techniques like grid search and Tree-structured parzen estimators. Even though it was written in order to show pros and cons of different methods for neural networks the main knowledge is generalizable for any other machine learning domains

$\endgroup$
  • 1
    $\begingroup$ I just wanted to point out that in your first sentence, the actual reason for why Bayesian optimization is better is not that the decisions being made are smarter, it is about decisions being made at all. $\endgroup$ – Alexander Rossa Apr 30 '18 at 23:10
1
$\begingroup$

Of note, Bayesian hyperparameter optimization is a sequential process, so it may take longer than some other approaches able to search or be conducted in parallel.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.