# What's your methodology of tuning neural network hyperparameters?

I'm curious to see what methods all of you use to tune your neural network hyperparameters. Specifically,

• How do you choose the number of layers, and how many hidden units are in each layer?
• Do you use manual parameter tuning, or random search, grid search, or Bayesian methods?
• How do you tune the optimizer, learning rate, regularization methods, etc?

Any wisdom would be greatly appreciated.

• I don't think that you'll get a definitive answer here, because none exist. What might be useful here would be some sort of community wiki where people answer the question "how do YOU tune your hyperparameters, and why?" – generic_user Jun 16 '17 at 19:00
• @generic_user you're right, I'll edit the question to reflect that. Would you mind being the first to answer :-)? Thanks. – hyperdo Jun 16 '17 at 19:13
• @generic_user how does that look? – hyperdo Jun 16 '17 at 19:18
• OK, will do. I have actually wanted to see a thread like this because I have been curious what others do, and only encounter threads saying things like "there are no good rules of thumb". – generic_user Jun 16 '17 at 19:27

I'm typically searching across architectures. Within each architecture, I implement something like the following algorithm

1. Initialize L2 penalty AKA weight decay parameter $\lambda$ at something large. Initialize parameters
2. Train network to approx convergence
3. Check performance against test set, save OOS error to vector OOS_err_vec
4. If the last 3 or 4 entries of OOS_err_vec are greater than the minimum of OOS_err_vec return the network that reached the minimum. If not:
5. $\lambda \leftarrow \lambda \div 2$, go to step 2, starting with parameters from last network

• It seems to work fairly well, most of the time
• Convergence is quick with a large $\lambda$, which is where you start
• You don't need to start from scratch when trying a new value of $\lambda$

• Starting with heavy penalization can set all your parameters to something close to zero, and this can mean that they get "stuck". Using ReLUs or leaky ReLUs helps. I've found however that starting from high $\lambda$ and halving tends to work better than starting from low $\lambda$ and then doubling -- I get to lower values of the loss function for any given $\lambda$, and OOS performance is better.