# hyperparameter tuning in neural networks

I was trying to fine tune a neural network model for a multilabel classification problem. I was reading Jason Brownlee 's article for the same. As per the article, there are a number of parameters to optimize which are:

1. batch size and training epochs
2. optimization algorithm
3. learning rate and momentum
4. network weight initialization
5. activation function in the hidden layer
6. dropout regularization
7. the number of neurons in the hidden layer

The code snippet is as below.

model = KerasClassifier(build_fn=create_model, verbose=1)
# define the grid search parameters
batch_size = [10, 20, 40, 60, 80, 100]
epochs = [10, 50, 100]
learn_rate = [0.001, 0.01, 0.1, 0.2, 0.3]
momentum = [0.0, 0.2, 0.4, 0.6, 0.8, 0.9]
weight_constraint = [1, 2, 3, 4, 5]
dropout_rate = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
neurons = [1, 5, 10, 15, 20, 25, 30]

param_grid = dict(neurons=neurons, batch_size=batch_size, epochs=epochs, learn_rate=learn_rate,
momentum=momentum, dropout_rate=dropout_rate, weight_constraint=weight_constraint)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1)
grid_result = grid.fit(X_train, y_train, validation_split=0.2)


Along with this, the number of hidden layers in the network is also another parameter.

I was doing hold out partitioning of the data and grid search for fine tuning. But it is taking huge time for computation even in a GPU machine.

Here I specified all these parameters in the same grid. I was wondering can we simplify this probably by finding the each parameter separately? For example, finding the optimal number of neurons first then, finding the batch size, etc.. What other approaches could be followed to reduce the search time?

I was also reading Bengio's paper Practical Recommendations for Gradient-Based Training of Deep Architectures but could not get much.

• Closely related: stats.stackexchange.com/questions/193306/… – Sycorax Apr 24 '18 at 14:34
• As you've discovered, a grid search over all possible configuration choices is usually too expensive to be exhaustive. Most people treat some aspects of the network as fixed (such as the activation function, initializer and optimizer) and only sequentially tune the others (for example, starting with a very small number of hidden neurons and then finding a good combination of learning rate, batch size and momentum before increasing the number of neurons and re-tuning the learning rate). – Sycorax Apr 24 '18 at 14:36

The link provided in @itdxer's comment is great. Based on this link, I am writing this answer. Hyperparameter optimization is neural networks is a tedious job as it contains many set of parameters.

The possible approaches for finding the optimal parameters are:

1. Hand tuning (Trial and Error) - @Sycorax's comment provides an example of hand tuning. Here, based on trial and error experiments and experience of the user, parameters are chosen.
2. Grid Search - Here a grid is created based on parameter values. And then all possible parameter combinations is tried and and the best one is selected.
3. Random Search - Here, instead of trying all possible combinations as in Grid Search, only randomly selected subset of the parameters is tried and the best is chosen.
4. Bayesian Optimization (Gausian Proces) - Gaussian Process uses a set of previously evaluated parameters and resulting accuracy to make an assumption about unobserved parameters. Acquisition Function using this information suggest the next set of parameters. (Do not understand much, taken from this link)
5. Tree-structured Parzen Estimators (TPE) - Each iteration TPE collects new observation and at the end of the iteration, the algorithm decides which set of parameters it should try next. (Do not understand much, taken from this link).