When reading about methods of hyperparameter tuning of neural networks, I have mainly come across grid search and random search in textbooks and articles online. I was wondering why other optimization methods such as Simulated Annealing or Genetic Algorithms not used? Are there any benefits to random search over the others?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Random search it trivial to implement, that's why it is so popular. Moreover, there are mixed results on comparing efficiency of different hyperparameter tuning algorithms: in many cases more advanced approaches like Bayesian optimization (based on Gaussian process, or Tree Parzen Estimator) work better, but there are also results showing that random search gives comparable results, or is not much worse, especially if you double the number of iterations (for discussion see this blog entry by Kevin Jamieson and Ben Recht and this PyData Berlin 2018 talk by Thorben Jensen). Long story short, if you want something that is "not bad" for hyperparameter tuning, but you don't want to bother that much about technicalities, then random search is one of the possible choices.
$\begingroup$
$\endgroup$
Other benefits of random search in addition to what Tim mentioned are
- Trivially parallelizable -- if it takes two days to train your model, and the deadline is in 3 days, random search works perfectly fine. Other sequential search algorithms may not.
- No "hyperhyperparameters" to tune, besides specifying search range. Many other hyperparameter search algorithms have hyperparameters themselves.
