# Machine learning cookbook / reference card / cheatsheet?

I find resources like the Probability and Statistics Cookbook and The R Reference Card for Data Mining incredibly useful. They obviously serve well as references but also help me to organize my thoughts on a subject and get the lay of the land.

Q: Does anything like these resources exist for machine learning methods?

I'm imagining a reference card which for each ML method would include:

• General properties
• When the method works well
• When the method does poorly
• From which or to which other methods the method generalizes. Has it been mostly superseded?
• Seminal papers on the method
• Open problems associated with the method
• Computational intensity

All these things can be found with some minimal digging through textbooks I'm sure. It would just be really convenient to have them on a few pages.

• A nice goal, but "minimal digging through some textbooks" ? How could one even start to compress say these 20 Books for statistical learning and data mining + mloss.org/software/rating ? – denis Jun 27 '11 at 10:48
• another one: stats.stackexchange.com/questions/19311/… – steffen Dec 20 '11 at 16:10
• (+1) for the chuzpa, if such an overview would exist, I'd pay for it. The key problem is that beside some properties which could be derived from the algorithm itself, the majority of such properties or rules of the thumb is gained by experience, i.e. application. I am pretty sure a battle-hardened applied researcher or ML-framework-programmer/consultant could write something like that ... but here and now ? – steffen Jun 6 '12 at 7:20
• @Denis: the "20 books.." link does not work, can you check this? – lmsasu Jul 26 '12 at 11:27
• I'm no machine learning expert so I will defer to others to post answers but I do think that The Elements of Statistical Learning is considered a good text on the subject and is written by some of the biggest names in the field. I should add that this book is written at a high level and those I've heard recommend it did have PhDs in statistics. – Macro Sep 1 '12 at 20:06

Some of the best and freely available resources are:

As to the author's question I haven't met "All in one page" solution

• Sergey, is Barber's book tied to Matlab ? – denis Jun 28 '11 at 9:19
• Yes, Just take a look at the book's link: The BRMLtool box is provided to help readers see how mathematical models translate into actual MAT-LAB code. – Sergey Jun 28 '11 at 9:26

If you want to learn Machine Learning I strongly advise you enroll in the free online ML course in the winter taught by Prof. Andrew Ng.

I did the previous one in the autumn and all learning material is of exceptional quality and geared toward practical applications, and a lot easier to grok that struggling alone with a book.

It's also made a pretty low hanging fruit with good intuitive explanations and the minimum amount of math.

• I just finished this course and it is awesome! Also, it gave me a great start to understanding books on machine learning. – B Seven Dec 20 '11 at 17:07
• I think this link now is coursera.org/course/ml – n611x007 May 2 '13 at 7:15

Yes, you are fine; Christopher Bishop's "Pattern Recognition and Machine Learning" is an excellent book for general reference, you can't really go wrong with it.

A fairly recent book but also very well-written and equally broad is David Barber's "Bayesian Reasoning and Machine Learning"; a book I would feel is slightly more suitable for a new-comer in the field.

I have used "The Elements of Statistical Learning" from Hastie et al. (mentioned by Macro) and while a very strong book I would not recommended it as a first reference; maybe it would serve you better as a second reference for more specialized topics. In that aspect, David MacKay's book, Information Theory, Inference, and Learning Algorithms, can also do a splendid job.

• +1 for Bishop. Clear development with an even level of detail. While still good I always found Hastie et al. a little bit choppy. – conjugateprior Sep 1 '12 at 20:54
• +1 -- Hastie , Tibshirani and Friedman is my personal favorite. – StasK Sep 1 '12 at 23:16
• +1 too for recommending Hastie, Tibshirani and Friedman, my personal favorite too. And thanks for the other recommendations; I'll give them a read because I really need a good book to recommend to non-statistician (or persons just entering the field). – Néstor Sep 2 '12 at 20:43
• +1 for Bishop. It's actually a great source for classical stats too, but updated and in disguise. – conjectures May 29 '14 at 19:28

Since the consensus seems to be that this question is not a duplicate, I'd like to share my favorite for machine learner beginners:

I found Programming Collective Intelligence the easiest book for beginners, since the author Toby Segaran is is focused on allowing the median software developer to get his/her hands dirty with data hacking as fast as possible.

Typical chapter: The data problem is clearly described, followed by a rough explanation how the algorithm works and finally shows how to create some insights with just a few lines of code.

The usage of python allows one to understand everything rather fast (you do not need to know python, seriously, I did not know it before, too). DONT think that this book is only focused on creating recommender system. It also deals with text mining / spam filtering / optimization / clustering / validation etc. and hence gives you a neat overview over the basic tools of every data miner.

Witten and Frank, "Data Mining", Elsevier 2005 is a good book for self-learning as there is a Java library of code (Weka) to go with the book and is very practically oriented. I suspect there is a more recent edition than the one I have.

• Yes, this book was to be called "Machine Learning" but the name was changed to "Data Mining" by the publishers to ride on the data mining hype at the time, nevertheless the book is about ML not DM (the two bare similarities, but are different fields!). – clyfe Dec 20 '11 at 16:03
• Tom Mitchell's book "Machine Learning" is also very good; the style is a bit old-fasioned, but the content is excellent. – Dikran Marsupial Dec 20 '11 at 16:15
• Yes, Tom Mitchell's ML is like the ML bible, really comprehensive on the field! – clyfe Dec 20 '11 at 20:13

I have Machine Learning: An Algorithmic Perspective by Stephen Marsland and find it very useful for self-learning. Python code is given throughout the book.

I agree with what is said in this favourable review:

http://blog.rtwilson.com/review-machine-learning-an-algorithmic-perspective-by-stephen-marsland/

"Elements of Statistical Learning" would be a great book for your purposes. The 5th edition of the book, which was published in early 2011, is freely available at http://www.stanford.edu/~hastie/local.ftp/Springer/ESLII_print5.pdf

• it is mathematics heavy book therefore it may be difficult for self learner to follow. – Atilla Ozgur Dec 24 '11 at 13:22
• Do you know how it happens to be freely downloadable on Trevor Hastie's personnal pages when Springer charges 70\$ for it? – Alfred M. Jul 27 '12 at 19:11
• I don't know for sure, but I would imagine Springer wants the money, and the authors mainly want to publicize their book widely. This seems very similar to how Springer will sell you published articles while many "working paper versions" are freely available on the author's website. – DanB Jul 29 '12 at 14:20
• FYI, the download is for the 5th printing of the second edition. I love the footnote to the eipgraph "In God we trust, all others bring data" which is attributed to Deming. The footnote points out the irony that no "data" can be found confirming Deming actually saying this. – HeatfanJohn Sep 25 '12 at 16:29
• You should mention Introduction to Statistical Learning with R--it's sort of like their ESL-lite (if the math in ESL is too daunting). – Steve S Nov 15 '14 at 11:19

Often the hardest part of solving a machine learning problem can be finding the right estimator for the job. Different estimators are better suited for different types of data and different problems. The flowchart below is designed to give users a bit of a rough guide on how to approach problems with regard to which estimators to try on your data. Click on any estimator in the chart below to see it’s documentation.

The awesome-machine-learning repository seems to be a master list of resources, including code, tutorials and books.

Most books mentioned in other answers are very good and you can't really go wrong with any of them. Additionally, I find the following cheat sheet for Python's scikit-learn quite useful.

I like Duda, Hart and Stork "Pattern Classification". This is a recent revision of a classic text that explains everything very well. Not sure that it is updated to have much coverage of neural networks and SVMs. The book by Hastie, Tibshirani and Friedman is about the best there is but may be a bit more technical than what you are looking for and is detailed rather than an overview of the subject.

Microsoft Azure also provides a similar cheat-sheet to the scikit-learn one posted by Anton Tarasenko.

They accompany it with a notice:

The suggestions offered in this algorithm cheat sheet are approximate rules-of-thumb. Some can be bent, and some can be flagrantly violated. This is intended to suggest a starting point. (...)

Microsoft additionally provides an introductory article providing further details.

Please notice that those materials are focused on the methods implemented in Microsoft Azure.

Don't start with Elements of Statistical Learning. It is great, but it is a reference book, which doesn't sound like what you are looking for. I would start with Programming Collective Intelligence as it's an easy read.

• I'm not sure I would characterize ESL as a reference text. It seems more of an overview to me, i.e., you aren't going to learn the nitty gritty details of (hardly) anything. You will see the broad techniques and overarching themes. – cardinal Sep 3 '12 at 14:11

For a first book on machine learning, which does a good job of explaining the principles, I would strongly recommend

Rogers and Girolami, A First Course in Machine Learning, (Chapman & Hall/CRC Machine Learning & Pattern Recognition), 2011.

Chris Bishop's book, or David Barber's both make good choices for a book with greater breadth, once you have a good grasp of the principles.

I wrote a summary like that, but only on one machine learning task (Netflix Prize), and it has 195 pages: http://arek-paterek.com/book

Check this link featuring some free ebooks on machine learning : http://designimag.com/best-free-machine-learning-ebooks/. it might be useful for you.

A good cheatsheet is the one in Max Kuhn book Applied Predictive Modeling. In the book there is a good summary table of several ML learning models. The table is in appendix A page 549.