What are the other optimization problems in deep learning other than training neural nets?

Question

I understand training deep neural nets is an optimization problem, however, I do not understand what other problems can be in deep learning that involves optimization? In Deep Learning book by Ian Goodfellow et al., they wrote in start of chapter 8 that:

Of all the many optimization problems involved in deep learning, the most difficult is neural network training

Secondly, is hyper-parameter optimization is another optimization problem in deep learning or is it a part of training problem? Thirdly, selecting an optimal architecture can be an optimization problem?

Answer 1

Yes, both hyperparameter tuning and architecture selection are optimization problems. Whether these are actually less difficult than NN training is debatable -- I think there are as around many papers on new architectures than there are on optimization techniques.

Certainly, they are easier in the sense that a human can manually tune parameters and select an architecture which works reasonably well, but not select NN weights.

Optimizing deep graphical models such as deep boltzmann machines is probably a more difficult optimization problem than training a neural network, depending on whether you consider DBMs a type of neural network.