Looking for book recommendations for numerical optimization

Question

I was reading the answers and comments to this question: Why is Newton's method not widely used in machine learning?

and realised that I would like to learn a lot more about numerical optimization. I was especially interested to notice some quite divergent answers.

I see that the following book was recommended:

Nocedal, J., & Wright, S. (n.d.). Numerical optimization (Second edition.). Springer-Verlag. 2006. https://www.springer.com/us/book/9780387303031

However, my university library has only 1 copy and it is high demand.

I am looking for other books of a similar level, that would be particularly useful for applications in statistics and machine learning (not just deep learning, but including DL too)

[I have a BS in Physics and I'm currently on a doctoral programme in data science]

Answer 1

Another two books similar to Nocedal & Wright are

• Numerical Optimization, by Bonnans et al.
• Optimization Theory and Methods, by Sun & Yuan

But since you're looking for optimization methods applicable in data science and machine learning, keep in mind that the sheer size of models in this field usually requires stochastic versions of algorithms discussed in these books (generally some version of stochastic gradient descent). This field is still in flux, and new algorithms seem to be developed every other month, so there aren't any comprehensive textbooks available at the moment (at least to my knowledge). The closest thing I could think of would be this review article by Ruder.

Answer 2

Aside Nocedal & Wright (2006), as books of a similar level, I have found:

• "Iterative methods in Optimisation" by Kelley (1999) and
• "Optimization" by Lange (2013).

Both books are equally easy to follow too with N&W and cover standard numerical optimisation topics (KKT Theory, Newton's Method, Quasi-Newton, etc.) nicely. Kelley's book is free to download too.

Answer 3

It's great that you shared your books. Thanks a lot for it! I really like Numerical Optimization (Springer Series in Operations Research and Financial Engineering), which you already mentioned, but I would still recommend Numerical Recipes 3rd Edition: The Art of Scientific Computing to study. I also recommend reading other educational resources, for example, you can find on eduzaurus many short articles on the key "Numerical". I think you will find a lot of useful information there.