I understand training deep neural nets is an optimization problem, however, I do not understand what other problems can be in deep learning that involves optimization? In Deep Learning book by Ian Goodfellow et al., they wrote in start of chapter 8 that:
Of all the many optimization problems involved in deep learning, the most difficult is neural network training
Secondly, is hyper-parameter optimization is another optimization problem in deep learning or is it a part of training problem? Thirdly, selecting an optimal architecture can be an optimization problem?