What's your methodology of tuning neural network hyperparameters?

Question

I'm curious to see what methods all of you use to tune your neural network hyperparameters. Specifically,

• How do you choose the number of layers, and how many hidden units are in each layer?
• Do you use manual parameter tuning, or random search, grid search, or Bayesian methods?
• How do you tune the optimizer, learning rate, regularization methods, etc?

Any wisdom would be greatly appreciated.

Answer 1

I'm typically searching across architectures. Within each architecture, I implement something like the following algorithm

1. Initialize L2 penalty AKA weight decay parameter $\lambda$ at something large. Initialize parameters
2. Train network to approx convergence
3. Check performance against test set, save OOS error to vector OOS_err_vec
4. If the last 3 or 4 entries of OOS_err_vec are greater than the minimum of OOS_err_vec return the network that reached the minimum. If not:
5. $\lambda \leftarrow \lambda \div 2$, go to step 2, starting with parameters from last network

Good points about this approach:

• It seems to work fairly well, most of the time
• Convergence is quick with a large $\lambda$, which is where you start
• You don't need to start from scratch when trying a new value of $\lambda$

Bad points about this approach:

• Starting with heavy penalization can set all your parameters to something close to zero, and this can mean that they get "stuck". Using ReLUs or leaky ReLUs helps. I've found however that starting from high $\lambda$ and halving tends to work better than starting from low $\lambda$ and then doubling -- I get to lower values of the loss function for any given $\lambda$, and OOS performance is better.

I apply that algorithm across a grid of architectures. Typically I'll choose 64 total architectures, and fit each of them in parallel on a AWS instance with that many cores (one can save lots of money using spot instances in place of on-demand instances, and rarely do the prices spike. Just set up your AMI how you want it.)

Then, I have 64 fitted neural nets. I simply take the best few and average their predictions. Doing so turns a chore -- tuning hyperparameters -- into something that benefits you -- model averaging reduces variability. What's more, I lose outliers by setting to NA any model-wise predictions that are way beyond what their colleagues predict.

My context: I'm using this package on continuous-outcome, repeated-observation data, to do forecasting on a system where there isn't a lot of temporal autocorrelation in the data-generating process, in the sense that last period's outcome doesn't determine this period's outcome (though they are correlated).

I hope others post their approaches -- I'd be eager to pick up some ideas.