I was trying to fine tune a neural network model for a multilabel classification problem. I was reading Jason Brownlee 's article for the same. As per the article, there are a number of parameters to optimize which are:

  1. batch size and training epochs
  2. optimization algorithm
  3. learning rate and momentum
  4. network weight initialization
  5. activation function in the hidden layer
  6. dropout regularization
  7. the number of neurons in the hidden layer

The code snippet is as below.

model = KerasClassifier(build_fn=create_model, verbose=1)
# define the grid search parameters
batch_size = [10, 20, 40, 60, 80, 100]
epochs = [10, 50, 100]
learn_rate = [0.001, 0.01, 0.1, 0.2, 0.3]
momentum = [0.0, 0.2, 0.4, 0.6, 0.8, 0.9]
weight_constraint = [1, 2, 3, 4, 5]
dropout_rate = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
neurons = [1, 5, 10, 15, 20, 25, 30]

param_grid = dict(neurons=neurons, batch_size=batch_size, epochs=epochs, learn_rate=learn_rate, 
                 momentum=momentum, dropout_rate=dropout_rate, weight_constraint=weight_constraint)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1)
grid_result = grid.fit(X_train, y_train, validation_split=0.2)

Along with this, the number of hidden layers in the network is also another parameter.

I was doing hold out partitioning of the data and grid search for fine tuning. But it is taking huge time for computation even in a GPU machine.

Here I specified all these parameters in the same grid. I was wondering can we simplify this probably by finding the each parameter separately? For example, finding the optimal number of neurons first then, finding the batch size, etc.. What other approaches could be followed to reduce the search time?

I was also reading Bengio's paper Practical Recommendations for Gradient-Based Training of Deep Architectures but could not get much.

  • 2
    $\begingroup$ This article might help you: neupy.com/2016/12/17/… $\endgroup$
    – itdxer
    Apr 24, 2018 at 12:30
  • $\begingroup$ Closely related: stats.stackexchange.com/questions/193306/… $\endgroup$
    – Sycorax
    Apr 24, 2018 at 14:34
  • 2
    $\begingroup$ As you've discovered, a grid search over all possible configuration choices is usually too expensive to be exhaustive. Most people treat some aspects of the network as fixed (such as the activation function, initializer and optimizer) and only sequentially tune the others (for example, starting with a very small number of hidden neurons and then finding a good combination of learning rate, batch size and momentum before increasing the number of neurons and re-tuning the learning rate). $\endgroup$
    – Sycorax
    Apr 24, 2018 at 14:36

2 Answers 2


The link provided in @itdxer's comment is great. Based on this link, I am writing this answer. Hyperparameter optimization is neural networks is a tedious job as it contains many set of parameters.

The possible approaches for finding the optimal parameters are:

  1. Hand tuning (Trial and Error) - @Sycorax's comment provides an example of hand tuning. Here, based on trial and error experiments and experience of the user, parameters are chosen.
  2. Grid Search - Here a grid is created based on parameter values. And then all possible parameter combinations is tried and and the best one is selected.
  3. Random Search - Here, instead of trying all possible combinations as in Grid Search, only randomly selected subset of the parameters is tried and the best is chosen.
  4. Bayesian Optimization (Gausian Proces) - Gaussian Process uses a set of previously evaluated parameters and resulting accuracy to make an assumption about unobserved parameters. Acquisition Function using this information suggest the next set of parameters. (Do not understand much, taken from this link)
  5. Tree-structured Parzen Estimators (TPE) - Each iteration TPE collects new observation and at the end of the iteration, the algorithm decides which set of parameters it should try next. (Do not understand much, taken from this link).

Now as per this link

The Bayesian Optimization and TPE algorithms show great improvement over the classic hyperparameter optimization methods. They allow to learn from the training history and give better and better estimations for the next set of parameters.

Now the good thing is that there is a Python library called hyperopt for doing these.

More details in the below pages:


https://jaberg.github.io/hyperopt/ .

https://github.com/jaberg/hyperopt/wiki .




You need only replace your grid search with a different global search. Recommended steps:

  • Google "optimizer elo ratings" to get a rough idea what works well
  • From the well-performing algorithms, click through to the packages to see which already have integration with keras, tensorflow or similar.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.