2
$\begingroup$

Setting hyper-parameters in Deep learning models is considered more of intuitive or some form of black art.

Hyper-Parameter Optimization (HPO) methods paves a principled approach of finding it.

Apart from providing a principled approach, why HPO should be considered seriously? In practice, does HPO makes any significant difference and how it offsets the time that has been invested in doing HPO?

Are their any real-life examples where HPO played a significant role in improving the Deep Learning models?

$\endgroup$
2
  • $\begingroup$ (1) You've used capitals to write "Hyper-Parameter Optimization" and Deep Learning. Does this indicate that you have a specific algorithm or method in mind, or do you mean to speak generally about hyperparameter search? (2) To answer your question, take a look at regularization hyperparameters in the context of overfitting. $\endgroup$
    – Sycorax
    Jul 28, 2017 at 19:59
  • $\begingroup$ Not any single method but in general. $\endgroup$ Jul 28, 2017 at 20:04

3 Answers 3

4
$\begingroup$

I would say the hyper-parameters are most important things in the model. For example, number of hidden layers and number of unites in hidden layer, and regularization parameters. These parameters will decide the model is over fitting or under fitting on a specific data set.

Check bias variance trade off here

An extreme example would be: if we set hidden layer to be 1 and 1 hidden unit, with regularization parameter 10 million. The model will provide nothing.

$\endgroup$
1
$\begingroup$

Of course HPO makes a difference. Take for example, the number of layers in a neural net. A neural net with 7 layers is much more likely to perform better on a large data set than say, a 1 layer neural network.

In terms of real life examples, if you look at the results from the Image Net competition, the winners of the competition are coming up with good combinations of hyperparameters for the # of neurons in a layer, # of layers, activation functions, etc. The difference between good hyperparameter optimization and poor HPO can make a huge impact in the accuracy/error.

$\endgroup$
0
$\begingroup$

HPO makes a huge difference. Over the past few years, many new architecture modifications have been proposed for deep learning. Often, authors show that their new modification performs slightly better than previous techniques. While many of these new methods do indeed improve the performance of deep learning models slightly, often, similar performance improvements can be achieved using proper HPO. In the paper of Isensee et al. this is illustrated by their "No new U-net". Isensee et al. shows here that using a basic U-net that is properly tuned outperforms all previously reported architecture modifications, achieving highest performance on the BRATS dataset. This in my opinion shows how important HPO is for achieving good performance.

If you'd like to read up on how to optimize hyperparameters in a deep learning model in an easy practical way I would suggest reading the paper of Smith - A disciplined approach to neural network hyper-parameters: part1 - learning rate, batch size, momentum, and weight decay.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.