Setting hyper-parameters in Deep learning models is considered more of intuitive or some form of black art.

Hyper-Parameter Optimization (HPO) methods paves a principled approach of finding it.

Apart from providing a principled approach, why HPO should be considered seriously? In practice, does HPO makes any significant difference and how it offsets the time that has been invested in doing HPO?

Are their any real-life examples where HPO played a significant role in improving the Deep Learning models?

  • $\begingroup$ (1) You've used capitals to write "Hyper-Parameter Optimization" and Deep Learning. Does this indicate that you have a specific algorithm or method in mind, or do you mean to speak generally about hyperparameter search? (2) To answer your question, take a look at regularization hyperparameters in the context of overfitting. $\endgroup$ – Sycorax says Reinstate Monica Jul 28 '17 at 19:59
  • $\begingroup$ Not any single method but in general. $\endgroup$ – letsBeePolite Jul 28 '17 at 20:04

I would say the hyper-parameters are most important things in the model. For example, number of hidden layers and number of unites in hidden layer, and regularization parameters. These parameters will decide the model is over fitting or under fitting on a specific data set.

Check bias variance trade off here

An extreme example would be: if we set hidden layer to be 1 and 1 hidden unit, with regularization parameter 10 million. The model will provide nothing.


Of course HPO makes a difference. Take for example, the number of layers in a neural net. A neural net with 7 layers is much more likely to perform better on a large data set than say, a 1 layer neural network.

In terms of real life examples, if you look at the results from the Image Net competition, the winners of the competition are coming up with good combinations of hyperparameters for the # of neurons in a layer, # of layers, activation functions, etc. The difference between good hyperparameter optimization and poor HPO can make a huge impact in the accuracy/error.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.