I'm curious to see what methods all of you use to tune your neural network hyperparameters. Specifically,

  • How do you choose the number of layers, and how many hidden units are in each layer?
  • Do you use manual parameter tuning, or random search, grid search, or Bayesian methods?
  • How do you tune the optimizer, learning rate, regularization methods, etc?

Any wisdom would be greatly appreciated.

  • $\begingroup$ I don't think that you'll get a definitive answer here, because none exist. What might be useful here would be some sort of community wiki where people answer the question "how do YOU tune your hyperparameters, and why?" $\endgroup$ Commented Jun 16, 2017 at 19:00
  • $\begingroup$ @generic_user you're right, I'll edit the question to reflect that. Would you mind being the first to answer :-)? Thanks. $\endgroup$
    – hyperdo
    Commented Jun 16, 2017 at 19:13
  • $\begingroup$ @generic_user how does that look? $\endgroup$
    – hyperdo
    Commented Jun 16, 2017 at 19:18
  • $\begingroup$ OK, will do. I have actually wanted to see a thread like this because I have been curious what others do, and only encounter threads saying things like "there are no good rules of thumb". $\endgroup$ Commented Jun 16, 2017 at 19:27

1 Answer 1


I'm typically searching across architectures. Within each architecture, I implement something like the following algorithm

  1. Initialize L2 penalty AKA weight decay parameter $\lambda$ at something large. Initialize parameters
  2. Train network to approx convergence
  3. Check performance against test set, save OOS error to vector OOS_err_vec
  4. If the last 3 or 4 entries of OOS_err_vec are greater than the minimum of OOS_err_vec return the network that reached the minimum. If not:
  5. $\lambda \leftarrow \lambda \div 2$, go to step 2, starting with parameters from last network

Good points about this approach:

  • It seems to work fairly well, most of the time
  • Convergence is quick with a large $\lambda$, which is where you start
  • You don't need to start from scratch when trying a new value of $\lambda$

Bad points about this approach:

  • Starting with heavy penalization can set all your parameters to something close to zero, and this can mean that they get "stuck". Using ReLUs or leaky ReLUs helps. I've found however that starting from high $\lambda$ and halving tends to work better than starting from low $\lambda$ and then doubling -- I get to lower values of the loss function for any given $\lambda$, and OOS performance is better.

I apply that algorithm across a grid of architectures. Typically I'll choose 64 total architectures, and fit each of them in parallel on a AWS instance with that many cores (one can save lots of money using spot instances in place of on-demand instances, and rarely do the prices spike. Just set up your AMI how you want it.)

Then, I have 64 fitted neural nets. I simply take the best few and average their predictions. Doing so turns a chore -- tuning hyperparameters -- into something that benefits you -- model averaging reduces variability. What's more, I lose outliers by setting to NA any model-wise predictions that are way beyond what their colleagues predict.

My context: I'm using this package on continuous-outcome, repeated-observation data, to do forecasting on a system where there isn't a lot of temporal autocorrelation in the data-generating process, in the sense that last period's outcome doesn't determine this period's outcome (though they are correlated).

I hope others post their approaches -- I'd be eager to pick up some ideas.

  • $\begingroup$ I'd be curious how you decide your initial list of architectures -- do you set with a wide array of layer-size combinations, or do you do something else? Thanks! $\endgroup$
    – hyperdo
    Commented Jun 16, 2017 at 20:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.