I'm interested in learning more about nonparametric Bayesian (and related) techniques. My background is in computer science and though I have never taken a course on measure theory or probability theory, I have had a limited amount of formal training in probability and statistics. Can anyone recommend a readable introduction to these concepts to get me started?
For a really short introduction (seven page pdf), there's also this, intended to allow you to follow papers that use a bit of measure theory :
A Measure Theory Tutorial (Measure Theory for Dummies). Maya R. Gupta. Dept of Electrical Engineering, University of Washington, 2006. (archive.org copy)
The author gives some refs at the end and says "one of the friendliest books is Resnick’s, which teaches measure theoretic graduate level probability with the assumption that you do not have a B.A. in mathematics."
S. I. Resnick, A probability path, Birkhäuser, 1999. 453 pages.
After some research, I ended up buying this when I thought I needed to know something about measure-theoretic probability:
I haven't read much of it, however, as my personal experience is in accord with Stephen Senn's quip.
Personally, I've found Kolmogorov's original Foundations of the Theory of Probability to be fairly readable, at least compared to most measure theory texts. Although it obviously doesn't contain any later work, it does give you an idea of most of the important concepts (sets of measure zero, conditional expectation, etc.). It is also mercifully brief, at only 84 pages.
Outline of Lebesgue Theory: A Heuristic Introduction by Robert E. Wernikoff. For engineers this is easily the best introduction.
Jumping straight into non-parametric Bayesian analysis is quite a big first leap! Maybe get a bit of parametric Bayes under your belt first?
Three books which you may find useful from the Bayesian part of things are:
1) Probability Theory: The Logic of Science by E. T. Jaynes, Edited by G. L. Bretthorst (2003)
2) Bayesian Theory by Bernardo, J. M. and Smith, A. F. M. (1st ed 1994, 2nd ed 2007).
3) Bayesian Decision Theory J. O. Berger (1985)
A good place to see recent applications of Bayesian statistics is the FREE journal called Bayesian Analysis, with articles from 2006 to present.