In some sense this is a crosspost of mine from math.stackexchange, and I have the feeling that this site might provide a broad audience.

I am looking for a mathematical introduction to machine learning. Particularly, lots of literature that can be found is relatively imprecise and a lot of pages are spent without any content.

However, starting from such literature, I discovered the Coursera courses from Andrew Ng, the book of Bishop on pattern recognition and finally a book of Smola. Unfortunately, the book of Smola is only in draft state. In Smola's book even proofs can be found, which appeals to me. Bishop's book is already quite good, but a certain amount of rigor is missing.

In short: I am looking for a book like Smola's, that is, as precise and rigorous as possible and uses mathematical background (though short introductions are of course OK).

Any recommendations?

  • 2
    $\begingroup$ In the future please dont crosspost. $\endgroup$
    – Momo
    Mar 25, 2015 at 19:58
  • $\begingroup$ It looks like the question is unfinished - it breaks off after "and". $\endgroup$
    – J W
    Mar 25, 2015 at 20:30
  • $\begingroup$ sorry, somehow my edit vanished. $\endgroup$ Mar 25, 2015 at 20:44
  • 1
    $\begingroup$ you might want to explain why a mathematician wants to learn about machine learning (to find a job as data scientist/ to do research/ etc) which will help people point you in the right direction $\endgroup$
    – seanv507
    Mar 25, 2015 at 20:46
  • 1
    $\begingroup$ for data science I would argue you need basic statistics understanding (eg linear/logistic regression),experimental design-eg ab testing etc,and in addition an understanding of recommender system techniques $\endgroup$
    – seanv507
    Mar 25, 2015 at 22:58

4 Answers 4


For what you describe, I highly recommend "Foundations of Machine Learning" by Mohri et.al. It is an undergraduate text, but it is for really good undergraduates. It is readable and it is the only place I have found what I would call a mathematical definition of machine learning (pac and weak pac). It is worth reading for that reason alone. I also have a math Phd. I'm familiar with, and like, many of the books mentioned above. I'm particularly fond of ESL for a broad spectrum of techniques and ideas, but it's a statistics book with lots of mathematics.

  • 1
    $\begingroup$ Btw, I'm told that Schapire, in his thesis proved that weak PAC implies PAC. His proof amounts to the boosting technique, so it's a nice example of how a theoretical question led to a very practical result. $\endgroup$
    – meh
    Mar 31, 2015 at 23:14
  • $\begingroup$ Thanks, for your remarks. I think I will work with ESL later after working with Mohri's and Shalev-Shwartz's books $\endgroup$ Apr 1, 2015 at 9:59

I would recommend Elements of Statistical Learning (free PDF file). It has sufficient maths and a good introduction to all the relevant techniques - together with some insights on why the techniques work (and when they don't).

Also Introduction to Statistical Learning (which is more practical - how to do it in R). It has a course running statistical learning; you might find the lectures on YouTube (and again free PDF).

  • 3
    $\begingroup$ That is a very nice recommendation. In addition to this, I suggest "Learning from Data" from Yaser S. Abu-Mostafa. It is heavily theoretical but explains very clearly topics such as feasibility of learning and VC dimension. The are videos and slides available online. $\endgroup$
    – tiagotvv
    Mar 26, 2015 at 10:29
  • $\begingroup$ I second the suggestion "Learning from Data" from Yaser S. Abu-Mostafa. The book is very short but packed with valuable information. Much focus is indeed put on feasibility of learning and complexity. $\endgroup$ Mar 31, 2015 at 23:23

You will probably like Learning With Kernels by Schölkopf and Smola. Most of Schölkopf's work is mathematically rigorous.

That said, you are probably better off reading research papers instead of textbooks. Research papers contain full derivations and proofs of convergence, bounds on performance, etc. which are very often not included in textbooks. A good place to start is the Journal of Machine Learning, which is highly regarded and fully open access. I also recommend the proceedings of conferences like ICML, NIPS, COLT and IJCNN.

  • $\begingroup$ thanks for the hints with the journal. However, I fear that the journals are, so far, too advanced for me. Nevertheless, this migth be a valuable source for the future. $\endgroup$ Mar 25, 2015 at 20:42

I would suggest Understanding Machine Learning: From Theory to Algorithms by Shai Shalev-Shwartz. I admit that I read only small portions of it but I immediately noticed rigor with which author approached every problem and discussion.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.