I was reading the answers and comments to this question: Why is Newton's method not widely used in machine learning?

and realised that I would like to learn a lot more about numerical optimization. I was especially interested to notice some quite divergent answers.

I see that the following book was recommended:

Nocedal, J., & Wright, S. (n.d.). Numerical optimization (Second edition.). Springer-Verlag. 2006. https://www.springer.com/us/book/9780387303031

However, my university library has only 1 copy and it is high demand.

I am looking for other books of a similar level, that would be particularly useful for applications in statistics and machine learning (not just deep learning, but including DL too)

[I have a BS in Physics and I'm currently on a doctoral programme in data science]

  • $\begingroup$ I have found the "autonlab" tutorials by Andrew Moore quite useful. They cover some of these, but are more hands-on. $\endgroup$ Commented Jul 28, 2020 at 16:34

3 Answers 3


Another two books similar to Nocedal & Wright are

But since you're looking for optimization methods applicable in data science and machine learning, keep in mind that the sheer size of models in this field usually requires stochastic versions of algorithms discussed in these books (generally some version of stochastic gradient descent). This field is still in flux, and new algorithms seem to be developed every other month, so there aren't any comprehensive textbooks available at the moment (at least to my knowledge). The closest thing I could think of would be this review article by Ruder.


Aside Nocedal & Wright (2006), as books of a similar level, I have found:

Both books are equally easy to follow too with N&W and cover standard numerical optimisation topics (KKT Theory, Newton's Method, Quasi-Newton, etc.) nicely. Kelley's book is free to download too.


It's great that you shared your books. Thanks a lot for it! I really like Numerical Optimization (Springer Series in Operations Research and Financial Engineering), which you already mentioned, but I would still recommend Numerical Recipes 3rd Edition: The Art of Scientific Computing to study. I also recommend reading other educational resources, for example, you can find on eduzaurus many short articles on the key "Numerical". I think you will find a lot of useful information there.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.