# What is the best introductory Bayesian statistics textbook?

Which is the best introductory textbook for Bayesian statistics?

-
In the replies, please explain why you are recommending a book as "the best." – whuber Jan 25 '12 at 15:33
How can there be more than one answer to a question question phrased like this? – naught101 Jul 24 '12 at 5:44

John Kruschke released a book in mid 2011 called Doing Bayesian Data Analysis: A Tutorial with R and BUGS. (A second edition was released in Nov 2014: Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan.) It is truly introductory. If you want to walk from frequentist stats into Bayes though, especially with multilevel modelling, I recommend Gelman and Hill.

John Kruschke also has a website for the book that has all the examples in the book in BUGS and JAGS. His blog on Bayesian statistics also links in with the book.

-
@Amir's suggestion is a duplicate of this. (The full title of the book is "Doing Bayesian Data Analysis: A Tutorial with R and BUGS".) As a truly introductory book, I've +1'd each. – Wayne Mar 12 '12 at 21:10
updated the title and added a couple of related links. – Jeromy Anglim Mar 21 '12 at 22:49
I also vote for Kruschke's book. I've browsed most of the books listed in the answers and this is the one I found most clear. IMO, it is the most clear stats book I have read. It helps a lot that R code is available to match formulas with code. The author starts with very simple examples and builds on them. Very little background is needed. All reviews on Amazon are highly favorable. Hoff's book is my second favorite. – julieth Jul 29 '12 at 13:02
Haha, I like the book cover: "Why the happy puppies? (as if happy puppies needed justification!)" – Zhubarb Oct 25 '13 at 9:21
My vote also goes to Kruschke's 2010 book. In trying to learn Bayesian statistics, I tried several of them, and this one hit the mark. Hard. – Patrick Coulombe Jan 24 '14 at 0:49

My favorite is "Bayesian Data Analysis" by Gelman, et al.

-
This is an introductory book for people who have a decent amount of statistical background already. – John Salvatier Nov 20 '10 at 23:41
I started a PhD in Statistics 9 months ago and to be honest Gelman's BDA is still above me, so I wouldn't call it an introductory text! – Sean Jun 28 '12 at 21:37
@Shane, can you please to this answer explaining why it's your favourite? – naught101 Jul 24 '12 at 5:45
-1, because according to multiple comments and other answers, this isn't introductory. – naught101 Jul 24 '12 at 5:48
@naught101 so you downvote without knowing the book? – conjectures Jan 28 at 12:25

Another vote for Gelman et al., but a close second for me -- being of the learn-by-doing persuasion -- is Bayesian Computation with R.

-
Agree strongly. Both great books. Start with Bayesian Computation With R, then get Gelman et al. – PeterR Jul 20 '10 at 19:12
I agree with @PeterR. – M. Tibbits Oct 5 '10 at 13:49

Sivia and Skilling, Data analysis: a Bayesian tutorial (2ed) 2006 246p 0198568320 books.goo:

Statistics lectures have been a source of much bewilderment and frustration for generations of students. This book attempts to remedy the situation by expounding a logical and unified approach to the whole subject of data analysis. This text is intended as a tutorial guide for senior undergraduates and research students in science and engineering ...

I don't know the other recommendations though.

-
This book it excellent. It's short and practical. – John Salvatier Nov 19 '10 at 1:48
I think this is a much better introductory text than Gelman. – Sean Jun 28 '12 at 21:38

I am an electrical engineer and not a statistician. I spent alot of time to go through Gelman but I don't think one can refer to gelman as introductory at all. My bayesian-guru professor from Carnegie Mellon is agree with me on this. having the minimum knowledge of statistics and R and Bugs(as the easy way to DO something with Bayesian stat) Doing Bayesian Data Analysis: A Tutorial with R and BUGS is an amazing start. you can compare all offered books easily by their book cover!

-
This is a duplicate of @rosser's answer above. As a truly introductory book, I've +1'd each. – Wayne Mar 12 '12 at 21:09
Never thought I'd say anything nice about a book with puppies on the front and cyan section headers, but... this looks just great. – conjugateprior Mar 21 '12 at 22:57

The Gelman books are all excellent but not necessarily introductory in that they assume that you know some statistics already. Therefore they are an introduction to the Bayesian way of doing statistics rather than to statistics in general. I would still give them the thumbs up, however.

As an introductory statistics/econometrics book which takes a Bayesian perspective, I would recommend Gary Koop's Bayesian Econometrics.

-

My favourite first undergraduate text for bayesian statistics is by Bolstad, Introduction to Bayesian Statistics. If you're looking for something graduate level, this will be too elementary, but for someone who is new to statistics this is ideal.

-
I think it's good for anyone. :) – Michelle Jan 23 '12 at 1:19

"Bayesian Core: A Practical Approach to Computational Bayesian Statistics" by Marin and Robert, Springer-Verlag (2007)

-
Thanks for mentioning our book!!! – Xi'an Jan 22 '12 at 21:14
@Xi'an and gappy, please explain why this book can be recommended. For whom is it suitable? In what sense is it "best"? – whuber Jan 25 '12 at 15:34
I do not want to fall into self-promotion. Bayesian Core is a self-contained entry to Bayesian inference for the most common models and to computational methods (R codes provided). It requires some background in probability theory that may be too much for some readers... (It works well with our 4th and 5th year students in France.) – Xi'an Jan 26 '12 at 14:48

I thoroughly recommend the entertaining polemic "Probability Theory: The Logic of Science" by E.T. Jaynes.

This is an introductory text in the sense of not requiring (and in fact preferring) no previous knowledge of statistics, but it does eventually employ fairly sophisticated mathematics. Compared to most of the other answers provided, this book is not nearly as practical or easy to digest, rather it provides the philosophical bedrock to why you would want to employ Bayesian methods, and why not to use frequentist approaches. It is introductory in a historical and philosophical, but not pedagogical way.

-
This is a brilliant book about Bayesian thinking rather than applying Bayesian methods. I think this is a good companion text to something which goes more into how do Bayesian computations. – probabilityislogic Jul 24 '12 at 12:29
That's a good way of putting it. I think Sivia and Skilling is an ideal companion text for introducing the methods in practice (which has already been suggested in another answer). – Bogdanovist Jul 24 '12 at 22:22

For an introduction, I would recommend Probabilistic Programming & Bayesian Methods for Hackers by Cam Davidson-Pilon, freely available online.

From its description:

An intro to Bayesian methods and probabilistic programming from a computation/understanding-first, mathematics-second point of view.

It's highly visual, cuts straight to the value and backfills gritty details later, has lots of examples, has interactive code (in IPython Notebook).

-
I thought this online book was hard to follow / poorly written. – captain_ahab Apr 11 at 22:37

Coming from non-statistical background I found Introduction to Applied Bayesian Statistics and Estimation for Social Scientists quite informative and easy to follow.

-

Its focus isn't strictly on Bayesian statistics, so it lacks some methodology, but David MacKay's Information Theory, Inference, and Learning Algorithms made me intuitively grasp Bayesian statistics better than others - most do the how quite nicely, but I felt MacKay explained why better.

-
Like Sivia, this is very nice if you have some physics background and can be rough if not. Not a good guide to any kind of applied social statistics (for that use Gelman and Hill, or Gelman et al. above) but really great for prompting you to really think about the core issues. – conjugateprior Dec 6 '10 at 21:12
This was the first book for me many years ago. – Memming Apr 14 '12 at 22:26

If you're looking for an elementary text, i.e. one that doesn't have a calculus prerequisite, there's Don Berry's Statistics: A Bayesian Perspective.

-

I have read some parts of A First Course in Bayesian Statistical Methods by Peter Hoff, and I found it easy to follow. (Example R-code is provided throughout the text)

-

I've at least glanced at most of these on this list and none are as good as the new Bayesian Ideas and Data Analysis in my opinion.

Edit: It is easy to immediately begin doing Bayesian analysis while reading this book. Not just model the mean from a Normal distribution with known variance, but actual data analysis after the first couple of chapters. All code examples and data are on the book's website. Covers a decent amount of theory but the focus is applications. Lots of examples over a wide range of models. Nice chapter on Bayesian Nonparametrics. Winbugs, R, and SAS examples. I prefer it over Doing Bayesian Data Analysis (I have both). Most of the books on here (Gelman, Robert, ...) are not introductory in my opinion and unless you have someone to talk to you will probably be left with more questions then answers. Albert's book does not cover enough material to feel comfortable analyzing data different from what is presented in the book (again my opinion).

-
"Good" in what sense? – whuber Jan 25 '12 at 15:34
Good point. Good as in best introductory Bayesian textbook. I believe it to be 'better' than Bayesian Data Analysis with R by Albert and I found Bayesian Data Analysis by Gelman et al. to not suffice as an introduction. After learning some Bayesian material however, it is a good reference. – Glen Apr 9 '12 at 14:18
@Glen: but WHY do you believe it is best? Well written? Logical layout? Good examples? Is it restricted to a particular field? – naught101 Jul 24 '12 at 5:51
@Glen, thanks for the edit, +1 – naught101 Dec 10 '12 at 10:12

For complete beginners, try William Briggs Breaking the Law of Averages: Real-Life Probability and Statistics in Plain English

-
curious that that book has no reviews on Amazon – Neil McGuigan Jul 19 '10 at 23:54

Take a look at "The Bayesian Choice". It has the full package: foundations, applications and computation. Clearly written.

-

I simply must to include MCMC in Practice. It provides an excellent introduction to MCMC, perhaps not as general as other books mentioned, but excellent for gaining insight and intuition. I would recommend reading it after (or in parallel with) Bayesian Computation with R.

-
Mcmc should not be the focus of an introduction to bayesian statistics, in my opinion. I think rejection sampling is more appealling as a way to understand how bayesian learning works. Also, least squares is bayesian (as is maximum likehood), so it too represents a gentler introduction to bayesian statistics, compared to mcmc. – probabilityislogic Jan 23 '12 at 9:59
My view is that mcmc should be avoided and used as a last resort - it simply takes too long in most cases (though i deal with big data sets where everything is basically mle). mcmc is a "sledge hammer" to some degree. Also mcmc is an algorithm for numerical integration. Nothing more, nothing less. It should receive the same introductory treatment as other algorithms, such as the laplace method and quadratre. Otherwise people will develop a narrow view of what "bayesian statistics" is. – probabilityislogic Jan 23 '12 at 12:13

If you happen to come from the physical sciencies (physics/astronomy) I would recommend you Bayesian Logical Data Analysis for the Physical Sciences: A Comparative Approach with Mathematica® Support by Gregory (2006).

Although the "with Mathematica® Support" part of the title is there only for commercial issues (the usages of Mathematica code are very poor), the good thing about this book is that it is truly an introduction to the subject of probabilities and statistics. It even has some chapters on frequentist statistics. However, once you give it a shot, go for the book of Gelman et. al that a lot of people recommended you. Most of the material in the book of Gregory is taken lightly (if not, it wouldn't be an introduction): Gelman's book has been a truly re-awakening from Gregory's for me.

-

I found an excellent introduction in Gelman and Hill (2006) Data Analysis Using Regression and Multilevel/Hierarchical Models. (Other comments mention it, but it deserves to get upvoted on its own.)

-

Not strictly Bayesian Statistics as such, but I can strongly recommend "A First Course on Machine Learning" by Rogers and Girolami, which is essentially an introduction to Bayesian approaches to machine learning. Its very well structured and clear and aimed at students without a strong mathematical background. This means it is a pretty good first introduction to Bayesian ideas. There is also MATLAB/OCTAVE code which is a nice feature.

-

Bayesian Statistics for Social Scientists. Phillips, Lawrence D. (1973), Thomas Crowell & Co. It's very clear, very accessible, assumes no statistics knowledge, and, unlike Bolstad which I found dry, has some personality.

-

Gelman et al. is well-regarded but explictly intended for a graduate course. If you don't have substantial prior coursework in statistics, it is largely a waste.

-
+0, this would serve better as a comment to @Shane's answer. You might edit your answer to recommend a book that would better sever the stated purpose for those without "substantial prior coursework in statistics". – gung Jan 23 '12 at 3:45

If I had to choose a single text for a beginner, it would be

              Sivia DS and Skilling J (2006) book (see below).


Of all the books listed below it strives hardest to give an intuitive grasp of the essential ideas, but it still requires some mathematical sophistication from page 1.

Below is a list of Further Readings from my book, with comments on each publication.

Bernardo, JM and Smith, A, (2000) 4 . Bayesian Theory A rigorous account of Bayesin methods, with many real-world examples.

Bishop, C (2006) 5 . Pattern Recognition and Machine Learning. As the title suggests, this is mainly about machine learning, but it provides a lucid and comprehensive account of Bayesian methods.

Cowan G (1998) 6 . Statistical Data Analysis. An excellent non-Bayesian introduction to statistical analysis.

Dienes, Z (2008) 8 . Understanding Psychology as a Science: An Introduction to Scientiﬁc and Statistical Inference. Provides tutorial material on Bayes’ rule and a lucid analysis of the distinction between Bayesian and frequentist statistics.

Gelman A, Carlin J, Stern H, and Rubin D. (2003) 14 . Bayesian Data Analysis. A rigorous and comprehensive account of Bayesian analysis, with many real-world examples.

Jaynes E and Bretthorst G (2003) 18 . Probability Theory: The Logic of Science. The modern classic of Bayesian analysis. It is comprehensive and wise. Its discursive style makes it long (600 pages) but never dull,and it is packed ful l of insights.

Khan, S, 2012, Introduction to Bayes’ Theorem. Salman Khan’s online mathematics videos make a good introduction to various topics, including Bayes’ rule.

Lee PM (2004) 27 . Bayesian Statistics: An Introduction. A rigorous and comprehensive text with a strident Bayesian style.

MacKay DJC (2003) 28 . Information theory, inference, and learning algorithms. The modern classic on information theory. A very readable text that roams far and wide over many topics, almost all of which make use of Bayes’ rule.

Migon, HS and Gamerman, D (1999) 30. Statistical Inference: An Integrated Approach. A straightforward (and clearly laid out) account of inference, which compares Bayesian and non-Bayesian approaches. Despite being fairly advanced, the writing style is tutorial in nature.

Pierce JR (1980) 34 2nd Edition. An introduction to information theory: symbols, signals and noise. Pierce writes with an informal, tutorial style of writing, but does not ﬂinch from presenting the fundamental theorems of information theory.

Reza, FM (1961) 35 . An introduction to information theory. A more comprehensive and mathematical ly rigorous book than the Pierce book above, and should ideally be read only after ﬁrst reading Pierce’s more informal text.

Sivia DS and Skilling J (2006) 38 . Data Analysis: A Bayesian Tutorial. This is an excellent tutorial style introduction to Bayesian methods.

Spiegelhalter, D and Rice, K (2009) 36 . Bayesian statistics. Scholarpedia, 4(8):5230. http://www.scholarpedia.org/article/Bayesian_statistics A reliable and comprehensive summary of the current status of Bayesian statistics.

And, here is my book, published June 2013.

Bayes' Rule: A Tutorial Introduction to Bayesian Analysis, Dr James V Stone, ISBN 978-0956372840

Description: Discovered by an 18th century mathematician and preacher, Bayes' rule is a cornerstone of modern probability theory. In this richly illustrated book, a range of accessible examples are used to show how Bayes' rule is actually a natural consequence of commonsense reasoning. Bayes' rule is derived using intuitive graphical representations of probability, and Bayesian analysis is applied to parameter estimation using the MatLab programs provided. The tutorial style of writing, combined with a comprehensive glossary, makes this an ideal primer for the novice who wishes to become familiar with the basic principles of Bayesian analysis.

-

Gelman et al (2013). Bayesian Data Analysis. CRC Press LLC. 3rd ed.

Hoff, Peter D (2009). A First Course in Bayesian Statistical Methods. Springer Texts in Statistics.

Kruschke, Doing Bayesian Data Analysis: A Tutorial with R and Bugs, 2011. Academic Press / Elsevier.

and I think that the better one to start with is Kruschke's book. It's perfect for a first approach to Bayesian thinking: concepts are explained very clearly, there is not too much mathematics, and there are lots of nice examples!

Gelman et al. is a great book, but it is more advanced and I suggest to read it after the Kruschke's one.

Conversely, I did not like Hoff's book because it is an introductory book, but concepts (and Bayesian thinking) are not explained in a clear way. I suggest to pass over.

-

Bayesian Modeling Using WinBUGS by Ioannis Ntzoufras, 2009.

Not only does this book introduce Bayesian methods, but includes everything one needs to immediately start running and diagnosing Bayesian models from simple normal models to generalized linear models to Bayesian hierarchical models. Not impressed yet? An entire chapter on how to set up and run WinBUGS (also useful for OpenBUGS!). Pseudo code and R code side-by-side throughout the book. Everything is illustrated with hands on examples. Also included are a chapter on the predictive distribution and model checking, and a chapter on checking convergence.

It is such an excellent combination of theory, explanation, application, software instruction and code.

-

I am now reading : From Algorithm to Z-Scores: Probabilistic and Statistical Modeling in Computer Science by Norm Matloff, UC Davis, freely available for download.

My two cents.

-
Well, do you like it? Would you recommend it to others? Is it for math-savvy or math-phobic readers? – gung Sep 19 '14 at 12:05
I'm still reading Chapter 2, (Probability and Conditional Probability). A little math, nothing exoteric so far, and easy-to-read... I like the approach but is too early to draw a full opinion. I leeave you here the link to the book to download... let me know what you think: (heather.cs.ucdavis.edu/~matloff/132/PLN/ProbStatBook.pdf) – Fabio Sep 20 '14 at 17:36

I'd recommend the following:

Gelman et al (2013). Bayesian Data Analysis. CRC Press LLC. 3rd ed.

Hoff, Peter D (2009). A First Course in Bayesian Statistical Methods. Springer Texts in Statistics.

Kruschke, Doing Bayesian Data Analysis: A Tutorial with R and Bugs, 2011. Academic Press / Elsevier.

Albert, Jim (2009). Bayesian Computation with R. 2nd ed. Springer.

-

This book suggests it is aimed at entry level undergraduate level

Biostatistics: A Bayesian Introduction. By George G Woodsworth.