29
$\begingroup$

In some sense this is a crosspost of mine from math.stackexchange, and I have the feeling that this site might provide a broad audience.

I am looking for a mathematical introduction to machine learning. Particularly, lots of literature that can be found is relatively imprecise and a lot of pages are spent without any content.

However, starting from such literature, I discovered the Coursera courses from Andrew Ng, the book of Bishop on pattern recognition and finally a book of Smola. Unfortunately, the book of Smola is only in draft state. In Smola's book even proofs can be found, which appeals to me. Bishop's book is already quite good, but a certain amount of rigor is missing.

In short: I am looking for a book like Smola's, that is, as precise and rigorous as possible and uses mathematical background (though short introductions are of course OK).

Any recommendations?

Improve this question
$\endgroup$
12
  • 2
    $\begingroup$ In the future please dont crosspost. $\endgroup$
    – Momo
    Commented Mar 25, 2015 at 19:58
  • $\begingroup$ It looks like the question is unfinished - it breaks off after "and". $\endgroup$
    – J W
    Commented Mar 25, 2015 at 20:30
  • $\begingroup$ sorry, somehow my edit vanished. $\endgroup$
    – Quickbeam2k1
    Commented Mar 25, 2015 at 20:44
  • 1
    $\begingroup$ you might want to explain why a mathematician wants to learn about machine learning (to find a job as data scientist/ to do research/ etc) which will help people point you in the right direction $\endgroup$
    – seanv507
    Commented Mar 25, 2015 at 20:46
  • 1
    $\begingroup$ for data science I would argue you need basic statistics understanding (eg linear/logistic regression),experimental design-eg ab testing etc,and in addition an understanding of recommender system techniques $\endgroup$
    – seanv507
    Commented Mar 25, 2015 at 22:58

4 Answers 4

Reset to default
15
$\begingroup$

For what you describe, I highly recommend "Foundations of Machine Learning" by Mohri et.al. It is an undergraduate text, but it is for really good undergraduates. It is readable and it is the only place I have found what I would call a mathematical definition of machine learning (pac and weak pac). It is worth reading for that reason alone. I also have a math Phd. I'm familiar with, and like, many of the books mentioned above. I'm particularly fond of ESL for a broad spectrum of techniques and ideas, but it's a statistics book with lots of mathematics.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Btw, I'm told that Schapire, in his thesis proved that weak PAC implies PAC. His proof amounts to the boosting technique, so it's a nice example of how a theoretical question led to a very practical result. $\endgroup$
    – meh
    Commented Mar 31, 2015 at 23:14
  • $\begingroup$ Thanks, for your remarks. I think I will work with ESL later after working with Mohri's and Shalev-Shwartz's books $\endgroup$
    – Quickbeam2k1
    Commented Apr 1, 2015 at 9:59
16
$\begingroup$

I would recommend Elements of Statistical Learning (free PDF file). It has sufficient maths and a good introduction to all the relevant techniques - together with some insights on why the techniques work (and when they don't).

Also Introduction to Statistical Learning (which is more practical - how to do it in R). It has a course running statistical learning; you might find the lectures on YouTube (and again free PDF).

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ That is a very nice recommendation. In addition to this, I suggest "Learning from Data" from Yaser S. Abu-Mostafa. It is heavily theoretical but explains very clearly topics such as feasibility of learning and VC dimension. The are videos and slides available online. $\endgroup$
    – tiagotvv
    Commented Mar 26, 2015 at 10:29
  • $\begingroup$ I second the suggestion "Learning from Data" from Yaser S. Abu-Mostafa. The book is very short but packed with valuable information. Much focus is indeed put on feasibility of learning and complexity. $\endgroup$
    – Vladislavs Dovgalecs
    Commented Mar 31, 2015 at 23:23
10
$\begingroup$

You will probably like Learning With Kernels by Schölkopf and Smola. Most of Schölkopf's work is mathematically rigorous.

That said, you are probably better off reading research papers instead of textbooks. Research papers contain full derivations and proofs of convergence, bounds on performance, etc. which are very often not included in textbooks. A good place to start is the Journal of Machine Learning, which is highly regarded and fully open access. I also recommend the proceedings of conferences like ICML, NIPS, COLT and IJCNN.

Improve this answer
$\endgroup$
1
  • $\begingroup$ thanks for the hints with the journal. However, I fear that the journals are, so far, too advanced for me. Nevertheless, this migth be a valuable source for the future. $\endgroup$
    – Quickbeam2k1
    Commented Mar 25, 2015 at 20:42
5
$\begingroup$

I would suggest Understanding Machine Learning: From Theory to Algorithms by Shai Shalev-Shwartz. I admit that I read only small portions of it but I immediately noticed rigor with which author approached every problem and discussion.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.