I have a graduate-level background in pure mathematics (Measure Theory, Functional Analysis, Operator Algebra, etc.) I also have a job that requires some knowledge of probability theory (from basic principles to machine learning techniques).

My question: Can someone provide some canonical reading and reference materials that:

  • Self-contained introduction to Probability theory
  • Don't shy away from measure theoretic methodologies and proofs
  • Provide a heavy emphasis on applied techniques.

Basically, I want a book that will teach me applied probability theory geared towards pure mathematicians. Something starting with the basic axioms of probability theory and introducing applied concepts with mathematical rigor.

As per the comments, I'll elaborate on what I need. I am doing basic-to-advanced data mining. Logistic Regression, Decision Trees, basic Stats and Probability (variance, standard deviation, likelihood, probability, likelihood, etc.), Supervised and Unsupervised machine learning (mainly clustering (K-Means, Hierarchal, SVM)).

With the above in mind, I want a book that will start at the beginning. Defining probability measures, but then also showing how those result in basic summation probabilities (which I know, intuitively, happen by integration over discrete sets). From there, it could go into: Markov Chains, Bayesian.... all the while discussing the foundational reasoning behind the theory, introducing the concepts with rigorous mathematics, but then showing how these methods are applied in the real world (specifically to data mining).

  1. Does such a book or reference exist?

Thank you!

PS - I realize this is similar in scope to this question. However, I'm looking for Probability theory and not statistics (as similar as the two fields are).

  • 2
    $\begingroup$ Can you expand briefly on what you mean by "applied techniques"? There are many excellent probability theory texts; e.g., Durrett's book is excellent for mathematicians that already know measure theory and it is loaded with examples. He doesn't hold your hand quite as much as other texts nor does he mind glossing over details in the proofs. This is actually nice for those with an already solid math background. $\endgroup$
    – cardinal
    Commented Jul 30, 2012 at 19:29
  • 1
    $\begingroup$ By applied I mean: I'm at work and I have to actually use probability theory. I have to be able to talk about basic things like the difference between "probability" and "likelihood" and things like that. Basically: imagine someone who has never learned any probability theory. But they also happen to be a mathematician who knows measure theory. $\endgroup$
    – aaronlevin
    Commented Jul 30, 2012 at 19:48
  • 2
    $\begingroup$ @aaronlevin, in my experience the field we refer to as "Applied Probability" is much more probability than applied. I like Applied Probability and Queues, with a concise treatment of Markov chains and other fundamental stochastic processes and with many illustrations of probabilistic models of Queues etc. However, I am not sure this is the probability book you are looking for. What kind of work do you do? By "applied" do you actually mean "statistics"? $\endgroup$
    – NRH
    Commented Jul 31, 2012 at 7:17
  • 1
    $\begingroup$ This question is a bit tricky, since "applied probability" could be any number of things. It would help if you told us a bit more about what kind of applications you have in mind. Algorithm analysis? Queue theory? Financial problems? Statistical physics? Telecommunications? Moreover, "likelihood" and "machine learning techniques" are parts of statistics more than they are part of probability theory. Very roughly, probability theory is concerned with modelling physical phenomenons, whereas statistics is concerned with inference from observations of those phenomenons. $\endgroup$
    – MånsT
    Commented Jul 31, 2012 at 10:19
  • 1
    $\begingroup$ Related: stats.stackexchange.com/a/7477/2970 $\endgroup$
    – cardinal
    Commented Jul 31, 2012 at 16:36

2 Answers 2


Though I am sure that @cardinal will also put together an excellent program, let me mention a couple of books that might cover some of the things the OP is asking for.

I recently came across Probability for Statistics and Machine Learning by Anirban DasGupta, which appears to me to cover many of the probabilistic topics asked for. It is fairly mathematical in its style, though it does not seem to be "hard core" measure theoretic. The best "hard core" books are, in my opinion, Real Analysis and Probability by Dudley and Foundations of Modern Probability by Kallenberg. These two very mathematical books should be accessible given the OPs background in functional analysis and operator algebra $-$ they may even be enjoyable. Neither of them has much to say about applications though.

On the more applied side I will definitely mention Elements of Statistical Learning by Hastie et al., which provides a treatment of many modern topics and applications from statistics and machine learning. Another book that I will recommend is In All Likelihood by Pawitan. It deals with more standard statistical material and applications and is fairly mathematical too.

  • 1
    $\begingroup$ (+1) Good suggestions! Thanks for taking the time to put these together. Kallenberg as a first encounter with probability theory, even for someone with background in measure theory, may be a tad on the ambitious side. Having Dudley (or any of several other texts) at hand would be sufficient, and, perhaps, necessary. $\endgroup$
    – cardinal
    Commented Aug 1, 2012 at 17:02

For a measure theory based introduction to probability I recommend Durrett's "Probability: Theory and Examples" (ISBN 0521765390) with Cosma Shalizi's "Almost None of the Theory of Stochastic Processes," (helpfully freely available http://www.stat.cmu.edu/~cshalizi/almost-none/v0.1.1/almost-none.pdf). I have not come across a perfect self contained book for everything after that. Some combination of MacKays's book (good for neural networks: http://www.inference.phy.cam.ac.uk/itprnn/book.html), the Koller and Friedman graphical models book (ISBN: 0262013193) and a good graduate level mathematical statistics book might work.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.