Which is the best introductory textbook for Bayesian statistics?
One book per answer, please.
John Kruschke released a book in mid 2011 called Doing Bayesian Data Analysis: A Tutorial with R and BUGS. (A second edition was released in Nov 2014: Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan.) It is truly introductory. If you want to walk from frequentist stats into Bayes though, especially with multilevel modelling, I recommend Gelman and Hill.
My favorite is "Bayesian Data Analysis" by Gelman, et al.
Statistical Rethinking, has been released just a few weeks ago and hence I am still reading it, but I think is a very nice and fresh addition to the really introductory books about Bayesian Statistics. The author uses a similar approach as the one used by John Kruschke in his puppy books; very verbose, detailed explanations, nice pedagogical examples, he also uses a computational rather than mathematical approach.
Youtube lectures and other material is also available from here.
Another vote for Gelman et al., but a close second for me -- being of the learn-by-doing persuasion -- is Jim Albert's "Bayesian Computation with R".
Sivia and Skilling, Data analysis: a Bayesian tutorial (2ed) 2006 246p 0198568320 books.goo:
Statistics lectures have been a source of much bewilderment and frustration for generations of students. This book attempts to remedy the situation by expounding a logical and unified approach to the whole subject of data analysis. This text is intended as a tutorial guide for senior undergraduates and research students in science and engineering ...
I don't know the other recommendations though.
For an introduction, I would recommend Probabilistic Programming & Bayesian Methods for Hackers by Cam Davidson-Pilon, freely available online.
From its description:
An intro to Bayesian methods and probabilistic programming from a computation/understanding-first, mathematics-second point of view.
It's highly visual, cuts straight to the value and backfills gritty details later, has lots of examples, has interactive code (in IPython Notebook).
I thoroughly recommend the entertaining polemic "Probability Theory: The Logic of Science" by E.T. Jaynes.
This is an introductory text in the sense of not requiring (and in fact preferring) no previous knowledge of statistics, but it does eventually employ fairly sophisticated mathematics. Compared to most of the other answers provided, this book is not nearly as practical or easy to digest, rather it provides the philosophical bedrock to why you would want to employ Bayesian methods, and why not to use frequentist approaches. It is introductory in a historical and philosophical, but not pedagogical way.
I am an electrical engineer and not a statistician. I spent a lot of time to go through Gelman but I don't think one can refer to Gelman as introductory at all. My bayesian-guru professor from Carnegie Mellon agrees with me on this. having the minimum knowledge of statistics and R and Bugs(as the easy way to DO something with Bayesian stat) Doing Bayesian Data Analysis: A Tutorial with R and BUGS is an amazing start. You can compare all offered books easily by their book cover!
5 years later update: I want to add that perhaps one other major way of learning in a fast way(40 mins) is to go through the documentation of a Bayesian Net GUI based tool such as Netica2. It starts with basics, walks you through the steps of building a net based on a situation and data, and how to run your own questions back and forth to "get it!".
Its focus isn't strictly on Bayesian statistics, so it lacks some methodology, but David MacKay's Information Theory, Inference, and Learning Algorithms made me intuitively grasp Bayesian statistics better than others - most do the how quite nicely, but I felt MacKay explained why better.
The Gelman books are all excellent but not necessarily introductory in that they assume that you know some statistics already. Therefore they are an introduction to the Bayesian way of doing statistics rather than to statistics in general. I would still give them the thumbs up, however.
As an introductory statistics/econometrics book which takes a Bayesian perspective, I would recommend Gary Koop's Bayesian Econometrics.
"Bayesian Core: A Practical Approach to Computational Bayesian Statistics" by Marin and Robert, Springer-Verlag (2007).
"Why?": the author explain the why of the bayesian choice and the how very well. It's a practical book, but written by one of the finest bayesian thinkers alive. It's not exhaustive. Other books have that objective. It picks up a few topics that are relevant, useful, and illuminating the foundations.
About "choice": if you really want to delve into bayesian foundation, Xi'an' "The Bayesian Choice" is clear, deep, essential.
My favourite first undergraduate text for bayesian statistics is by Bolstad, Introduction to Bayesian Statistics. If you're looking for something graduate level, this will be too elementary, but for someone who is new to statistics this is ideal.
I don't know why nobody has mentioned the very introductory book on Bayesian:
There's a free PDF version for the book. The book offers enough material for anyone who has very little experience in bayesian. It introduces the concept of prior distribution, posterior distribution, beta distribution etc.
Give it a go, it's free.
I have read some parts of A First Course in Bayesian Statistical Methods by Peter Hoff, and I found it easy to follow. (Example R-code is provided throughout the text)
I found an excellent introduction in Gelman and Hill (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models. (Other comments mention it, but it deserves to get upvoted on its own.)
Coming from non-statistical background I found Introduction to Applied Bayesian Statistics and Estimation for Social Scientists quite informative and easy to follow.
If you're looking for an elementary text, i.e. one that doesn't have a calculus prerequisite, there's Don Berry's Statistics: A Bayesian Perspective.
Take a look at "The Bayesian Choice". It has the full package: foundations, applications and computation. Clearly written.
I've at least glanced at most of these on this list and none are as good as the new Bayesian Ideas and Data Analysis in my opinion.
Edit: It is easy to immediately begin doing Bayesian analysis while reading this book. Not just model the mean from a Normal distribution with known variance, but actual data analysis after the first couple of chapters. All code examples and data are on the book's website. Covers a decent amount of theory but the focus is applications. Lots of examples over a wide range of models. Nice chapter on Bayesian Nonparametrics. Winbugs, R, and SAS examples. I prefer it over Doing Bayesian Data Analysis (I have both). Most of the books on here (Gelman, Robert, ...) are not introductory in my opinion and unless you have someone to talk to you will probably be left with more questions then answers. Albert's book does not cover enough material to feel comfortable analyzing data different from what is presented in the book (again my opinion).
I quite like Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference by Gamerman and Lopes.
For complete beginners, try William Briggs Breaking the Law of Averages: Real-Life Probability and Statistics in Plain English
I simply must to include MCMC in Practice. It provides an excellent introduction to MCMC, perhaps not as general as other books mentioned, but excellent for gaining insight and intuition. I would recommend reading it after (or in parallel with) Bayesian Computation with R.
If you happen to come from the physical sciencies (physics/astronomy) I would recommend you Bayesian Logical Data Analysis for the Physical Sciences: A Comparative Approach with Mathematica® Support by Gregory (2006).
Although the "with Mathematica® Support" part of the title is there only for commercial issues (the usages of Mathematica code are very poor), the good thing about this book is that it is truly an introduction to the subject of probabilities and statistics. It even has some chapters on frequentist statistics. However, once you give it a shot, go for the book of Gelman et. al that a lot of people recommended you. Most of the material in the book of Gregory is taken lightly (if not, it wouldn't be an introduction): Gelman's book has been a truly re-awakening from Gregory's for me.
Gelman et al (2013). Bayesian Data Analysis. CRC Press LLC. 3rd ed.
Hoff, Peter D (2009). A First Course in Bayesian Statistical Methods. Springer Texts in Statistics.
Kruschke, Doing Bayesian Data Analysis: A Tutorial with R and Bugs, 2011. Academic Press / Elsevier.
and I think that the better one to start with is Kruschke's book. It's perfect for a first approach to Bayesian thinking: concepts are explained very clearly, there is not too much mathematics, and there are lots of nice examples!
Gelman et al. is a great book, but it is more advanced and I suggest to read it after the Kruschke's one.
Conversely, I did not like Hoff's book because it is an introductory book, but concepts (and Bayesian thinking) are not explained in a clear way. I suggest to pass over.
If I had to choose a single text for a beginner, it would be
Sivia DS and Skilling J (2006) book (see below).
Of all the books listed below it strives hardest to give an intuitive grasp of the essential ideas, but it still requires some mathematical sophistication from page 1.
Below is a list of Further Readings from my book, with comments on each publication.
Bernardo, JM and Smith, A, (2000) 4 . Bayesian Theory A rigorous account of Bayesian methods, with many real-world examples.
Bishop, C (2006) 5 . Pattern Recognition and Machine Learning. As the title suggests, this is mainly about machine learning, but it provides a lucid and comprehensive account of Bayesian methods.
Cowan G (1998) 6 . Statistical Data Analysis. An excellent non-Bayesian introduction to statistical analysis.
Dienes, Z (2008) 8 . Understanding Psychology as a Science: An Introduction to Scientiﬁc and Statistical Inference. Provides tutorial material on Bayes’ rule and a lucid analysis of the distinction between Bayesian and frequentist statistics.
Gelman A, Carlin J, Stern H, and Rubin D. (2003) 14 . Bayesian Data Analysis. A rigorous and comprehensive account of Bayesian analysis, with many real-world examples.
Jaynes E and Bretthorst G (2003) 18 . Probability Theory: The Logic of Science. The modern classic of Bayesian analysis. It is comprehensive and wise. Its discursive style makes it long (600 pages) but never dull,and it is packed ful l of insights.
Khan, S, 2012, Introduction to Bayes’ Theorem. Salman Khan’s online mathematics videos make a good introduction to various topics, including Bayes’ rule.
Lee PM (2004) 27 . Bayesian Statistics: An Introduction. A rigorous and comprehensive text with a strident Bayesian style.
MacKay DJC (2003) 28 . Information theory, inference, and learning algorithms. The modern classic on information theory. A very readable text that roams far and wide over many topics, almost all of which make use of Bayes’ rule.
Migon, HS and Gamerman, D (1999) 30. Statistical Inference: An Integrated Approach. A straightforward (and clearly laid out) account of inference, which compares Bayesian and non-Bayesian approaches. Despite being fairly advanced, the writing style is tutorial in nature.
Pierce JR (1980) 34 2nd Edition. An introduction to information theory: symbols, signals and noise. Pierce writes with an informal, tutorial style of writing, but does not ﬂinch from presenting the fundamental theorems of information theory.
Reza, FM (1961) 35 . An introduction to information theory. A more comprehensive and mathematical ly rigorous book than the Pierce book above, and should ideally be read only after ﬁrst reading Pierce’s more informal text.
Sivia DS and Skilling J (2006) 38 . Data Analysis: A Bayesian Tutorial. This is an excellent tutorial style introduction to Bayesian methods.
Spiegelhalter, D and Rice, K (2009) 36 . Bayesian statistics. Scholarpedia, 4(8):5230. http://www.scholarpedia.org/article/Bayesian_statistics A reliable and comprehensive summary of the current status of Bayesian statistics.
And, here is my book, published June 2013.
Bayes' Rule: A Tutorial Introduction to Bayesian Analysis, Dr James V Stone, ISBN 978-0956372840
Chapter 1 can be downloaded from: http://jim-stone.staff.shef.ac.uk/BookBayes2012/BayesRuleBookMain.html
Description: Discovered by an 18th century mathematician and preacher, Bayes' rule is a cornerstone of modern probability theory. In this richly illustrated book, a range of accessible examples are used to show how Bayes' rule is actually a natural consequence of commonsense reasoning. Bayes' rule is derived using intuitive graphical representations of probability, and Bayesian analysis is applied to parameter estimation using the MatLab programs provided. The tutorial style of writing, combined with a comprehensive glossary, makes this an ideal primer for the novice who wishes to become familiar with the basic principles of Bayesian analysis.
Not strictly Bayesian Statistics as such, but I can strongly recommend "A First Course on Machine Learning" by Rogers and Girolami, which is essentially an introduction to Bayesian approaches to machine learning. Its very well structured and clear and aimed at students without a strong mathematical background. This means it is a pretty good first introduction to Bayesian ideas. There is also MATLAB/OCTAVE code which is a nice feature.
Gill, J. (2014). Bayesian Methods: A Social and Behavioral Sciences Approach. 3rd edition.
Written by a political science professor, with social scientists as the target audience in mind. R code is provided.
Since the type of beginner is not specified in the question, here is my advice for beginning statisticians:
Andrew B. Lawson and Emmanuel Lesaffre (2012): Bayesian Biostatistics
This book was used in the first year of our statistical science master and I found it relatively easy to understand for such a difficult subject. As with the majority of 'biostatistics' books, the examples are mainly clinical biology, but the methods are not restricted to those useful in clinical science. We had had about half a year of statistical education prior to this and besides Bayes theorem, Bayesian statistics hadn't been introduced yet.
What's also nice is that the entire 649 slides of accompanying presentations are available online.