I understand training deep neural nets is an optimization problem, however, I do not understand what other problems can be in deep learning that involves optimization? In Deep Learning book by Ian Goodfellow et al., they wrote in start of chapter 8 that:

Of all the many optimization problems involved in deep learning, the most difficult is neural network training

Secondly, is hyper-parameter optimization is another optimization problem in deep learning or is it a part of training problem? Thirdly, selecting an optimal architecture can be an optimization problem?


1 Answer 1


Yes, both hyperparameter tuning and architecture selection are optimization problems. Whether these are actually less difficult than NN training is debatable -- I think there are as around many papers on new architectures than there are on optimization techniques.

Certainly, they are easier in the sense that a human can manually tune parameters and select an architecture which works reasonably well, but not select NN weights.

Optimizing deep graphical models such as deep boltzmann machines is probably a more difficult optimization problem than training a neural network, depending on whether you consider DBMs a type of neural network.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.