I have read 2 papers (this and this) about the convergence of the Adam optimizer. One of the assumptions is the smoothness of the loss function, meaning that the gradient of the loss function is Lipschitz continuous. Let's consider a neural network $f(\theta)$ with a loss function $L$. If I want to prove the smoothness of the loss function, does it imply that I have to derive the calculation about the gradient of the loss function w.r.t every weight (do the backpropagation) and prove that the Lipschitz inequality holds?

Also, does this assumption depends on the network architecture?

  • 2
    $\begingroup$ most neural architectures do not admit lipschitz continuous loss (for example: a neural network with quadratic loss and 0 hidden layers (aka linear regression)); we use Adam on them anyways. (some analysis has tried to do away with this assumption, e.g. arxiv.org/abs/2206.13290). $\endgroup$ Commented May 20, 2023 at 15:05


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.