What's your methodology of tuning neural network hyperparameters?

I'm curious to see what methods all of you use to tune your neural network hyperparameters. Specifically,

• How do you choose the number of layers, and how many hidden units are in each layer?
• Do you use manual parameter tuning, or random search, grid search, or Bayesian methods?
• How do you tune the optimizer, learning rate, regularization methods, etc?

Any wisdom would be greatly appreciated.

• I don't think that you'll get a definitive answer here, because none exist. What might be useful here would be some sort of community wiki where people answer the question "how do YOU tune your hyperparameters, and why?" – generic_user Jun 16 '17 at 19:00
• @generic_user you're right, I'll edit the question to reflect that. Would you mind being the first to answer :-)? Thanks. – hyperdo Jun 16 '17 at 19:13
• @generic_user how does that look? – hyperdo Jun 16 '17 at 19:18
• OK, will do. I have actually wanted to see a thread like this because I have been curious what others do, and only encounter threads saying things like "there are no good rules of thumb". – generic_user Jun 16 '17 at 19:27

I'm typically searching across architectures. Within each architecture, I implement something like the following algorithm

1. Initialize L2 penalty AKA weight decay parameter $\lambda$ at something large. Initialize parameters
2. Train network to approx convergence
3. Check performance against test set, save OOS error to vector OOS_err_vec
4. If the last 3 or 4 entries of OOS_err_vec are greater than the minimum of OOS_err_vec return the network that reached the minimum. If not:
5. $\lambda \leftarrow \lambda \div 2$, go to step 2, starting with parameters from last network

• It seems to work fairly well, most of the time
• Convergence is quick with a large $\lambda$, which is where you start
• You don't need to start from scratch when trying a new value of $\lambda$

• Starting with heavy penalization can set all your parameters to something close to zero, and this can mean that they get "stuck". Using ReLUs or leaky ReLUs helps. I've found however that starting from high $\lambda$ and halving tends to work better than starting from low $\lambda$ and then doubling -- I get to lower values of the loss function for any given $\lambda$, and OOS performance is better.

I apply that algorithm across a grid of architectures. Typically I'll choose 64 total architectures, and fit each of them in parallel on a AWS instance with that many cores (one can save lots of money using spot instances in place of on-demand instances, and rarely do the prices spike. Just set up your AMI how you want it.)

Then, I have 64 fitted neural nets. I simply take the best few and average their predictions. Doing so turns a chore -- tuning hyperparameters -- into something that benefits you -- model averaging reduces variability. What's more, I lose outliers by setting to NA any model-wise predictions that are way beyond what their colleagues predict.

My context: I'm using this package on continuous-outcome, repeated-observation data, to do forecasting on a system where there isn't a lot of temporal autocorrelation in the data-generating process, in the sense that last period's outcome doesn't determine this period's outcome (though they are correlated).

I hope others post their approaches -- I'd be eager to pick up some ideas.

• I'd be curious how you decide your initial list of architectures -- do you set with a wide array of layer-size combinations, or do you do something else? Thanks! – hyperdo Jun 16 '17 at 20:40