Guideline to select the hyperparameters in Deep Learning

Question

I'm looking for a paper that could help in giving a guideline on how to choose the hyperparameters of a deep architecture, like stacked auto-encoders or deep believe networks. There are a lot of hyperparameters and I'm very confused on how to choose them. Also using cross-validation is not an option since training really takes a lot of time!

Answer 1

There are basically four methods:

1. Manual Search: Using knowledge you have about the problem guess parameters and observe the result. Based on that result tweak the parameters. Repeat this process until you find parameters that work well or you run out of time.
2. Grid Search: Using knowledge you have about the problem identify ranges for the hyperparameters. Then select several points from those ranges, usually uniformly distributed. Train your network using every combination of parameters and select the combination that performs best. Alternatively you can repeat your search on a more narrow domain centered around the parameters that perform the best.
3. Random Search: Like grid search you use knowledge of the problem to identify ranges for the hyperparameters. However instead of picking values from those ranges in a methodical manner you instead select them at random. Repeat this process until you find parameters that work well or use what you learn to narrow your search. In the paper Random Search for Hyper-Parameter Optimization Dr. Bengio proposes this be the baseline method against which all other methods should be compared and shows that it tends to work better than the other methods.
4. Bayesian Optimization: More recent work has been focus on improving upon these other approaches by using the information gained from any given experiment to decide how to adjust the hyper parameters for the next experiment. An example of this work would be Practical Bayesian Optimization of Machine Learning Algorithms by Adams et al.

Answer 2

A wide variety of methods exist. They can be largely partitioned in random/undirected search methods (like grid search or random search) and direct methods. Be aware, though, that they all require testing a considerable amount of hyperparameter settings unless you get lucky (hundreds at least, depends on the number of parameters).

In the class of direct methods, several distinct approaches can be identified:

• derivative free methods, for example the Nelder-Mead simplex or DIRECT
• evolutionary methods, such as CMA-ES and particle swarms
• model-based approaches, e.g. EGO and sequential Kriging

You may want to look into Optunity, a Python package which offers a variety of solvers for hyperparameter tuning (everything I mentioned except EGO and Kriging, for now). Optunity will be available for MATLAB and R soon. Disclaimer: I am the main developer of this package.

Based on my personal experience, evolutionary methods are very powerful for these types of problems.

Answer 3

Look no further! Yoshua Bengio published one of my favorite applied papers, one that I recommend to all new machine learning engineers when they start training neural nets: Practical recommendations for gradient-based training of deep architectures. To get his perspective on hyperparameter turning: including learning rate, learning rate schedule, early stopping, minibatch size, number of hidden layers, etc., see Section 3.