hyperparameter tuning in neural networks I was trying to fine tune a neural network model for a multilabel classification problem. I was reading Jason Brownlee 's  article for the same. As per the article, there are a number of parameters to optimize which are:


*

*batch size and training epochs

*optimization algorithm

*learning rate and momentum

*network weight initialization

*activation function in the hidden layer

*dropout regularization

*the number of neurons in the hidden layer


The code snippet is as below.
model = KerasClassifier(build_fn=create_model, verbose=1)
# define the grid search parameters
batch_size = [10, 20, 40, 60, 80, 100]
epochs = [10, 50, 100]
learn_rate = [0.001, 0.01, 0.1, 0.2, 0.3]
momentum = [0.0, 0.2, 0.4, 0.6, 0.8, 0.9]
weight_constraint = [1, 2, 3, 4, 5]
dropout_rate = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
neurons = [1, 5, 10, 15, 20, 25, 30]

param_grid = dict(neurons=neurons, batch_size=batch_size, epochs=epochs, learn_rate=learn_rate, 
                 momentum=momentum, dropout_rate=dropout_rate, weight_constraint=weight_constraint)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1)
grid_result = grid.fit(X_train, y_train, validation_split=0.2)

Along with this, the number of hidden layers in the network is also another parameter.
I was doing hold out partitioning of the data and grid search for fine tuning. But it is taking huge time for computation even in a GPU machine.
Here I specified all these parameters in the same grid. I was wondering can we simplify this probably by finding the each parameter separately? For example, finding the optimal number of neurons first then, finding the batch size, etc.. What other approaches could be followed to reduce the search time?
I was also reading Bengio's paper Practical Recommendations for Gradient-Based Training of Deep Architectures but could not get much.
 A: The link provided in @itdxer's comment is great. Based on this link, I am writing this answer. 
Hyperparameter optimization is neural networks is a tedious job as it contains many set of parameters. 
The possible approaches for finding the optimal parameters are:


*

*Hand tuning (Trial and Error) - @Sycorax's comment provides an example of hand tuning. Here, based on trial and error experiments and experience of the user, parameters are chosen.

*Grid Search - Here a grid is created based on parameter values. And then all possible parameter combinations is tried and and the best one is selected.

*Random Search - Here, instead of trying all possible combinations as in Grid Search, only randomly selected subset of the parameters is tried and the best is chosen.

*Bayesian Optimization (Gausian Proces) - Gaussian Process uses a set of previously evaluated parameters and resulting accuracy to make an assumption about unobserved parameters. Acquisition Function using this information suggest the next set of parameters. (Do not understand much, taken from this link)

*Tree-structured Parzen Estimators (TPE) - Each iteration TPE collects new observation and at the end of the iteration, the algorithm decides which set of parameters it should try next. (Do not understand much, taken from this link).


Now as per this link 

The Bayesian Optimization and TPE algorithms show great improvement
  over the classic hyperparameter optimization methods. They allow to
  learn from the training history and give better and better estimations
  for the next set of parameters.

Now the good thing is that there is a Python library called hyperopt for doing these.
More details in the below pages:
http://neupy.com/2016/12/17/hyperparameter_optimization_for_neural_networks.html 
https://jaberg.github.io/hyperopt/ . 
https://github.com/jaberg/hyperopt/wiki . 
https://github.com/wenyangfu/hyperparam-search-guides/blob/master/hyperopt-guide.md 
https://www.youtube.com/watch?v=Mp1xnPfE4PY
A: You need only replace your grid search with a different global search. Recommended steps:

*

*Google "optimizer elo ratings" to get a rough idea what works well

*From the well-performing algorithms, click through to the packages to see which already have integration with keras, tensorflow or similar.

