Which is the best introductory textbook for Bayesian statistics?

One book per answer, please.

  • 30
    In the replies, please explain why you are recommending a book as "the best." – whuber Jan 25 '12 at 15:33
  • 3
    How can there be more than one answer to a question question phrased like this? – naught101 Jul 24 '12 at 5:44
  • 5
    This is an old thread now, but I came back to +1 a new book "Statistical Rethinking. And in looking the higher-ranking answers in the thread, I think a key distinction hasn't been made: "introductory" for whom? A first course in statistics (that happens to have a Bayesian approach)? An introduction to Bayesian methods for someone with basic undergraduate (non-Bayesian) statistics classes? Or an introduction to Bayesian statistics for a practitioner of non-Bayesian statistics who has finally been persuaded that this Bayesian thing isn't a fad? Very different introductions. – Wayne Jan 20 '16 at 13:54

34 Answers 34

John Kruschke released a book in mid 2011 called Doing Bayesian Data Analysis: A Tutorial with R and BUGS. (A second edition was released in Nov 2014: Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan.) It is truly introductory. If you want to walk from frequentist stats into Bayes though, especially with multilevel modelling, I recommend Gelman and Hill.

John Kruschke also has a website for the book that has all the examples in the book in BUGS and JAGS. His blog on Bayesian statistics also links in with the book.

  • @Amir's suggestion is a duplicate of this. (The full title of the book is "Doing Bayesian Data Analysis: A Tutorial with R and BUGS".) As a truly introductory book, I've +1'd each. – Wayne Mar 12 '12 at 21:10
  • updated the title and added a couple of related links. – Jeromy Anglim Mar 21 '12 at 22:49
  • 3
    I also vote for Kruschke's book. I've browsed most of the books listed in the answers and this is the one I found most clear. IMO, it is the most clear stats book I have read. It helps a lot that R code is available to match formulas with code. The author starts with very simple examples and builds on them. Very little background is needed. All reviews on Amazon are highly favorable. Hoff's book is my second favorite. – julieth Jul 29 '12 at 13:02
  • Haha, I like the book cover: "Why the happy puppies? (as if happy puppies needed justification!)" – Berkan Oct 25 '13 at 9:21
  • My vote also goes to Kruschke's 2010 book. In trying to learn Bayesian statistics, I tried several of them, and this one hit the mark. Hard. – Patrick Coulombe Jan 24 '14 at 0:49

My favorite is "Bayesian Data Analysis" by Gelman, et al.

  • 23
    This is an introductory book for people who have a decent amount of statistical background already. – John Salvatier Nov 20 '10 at 23:41
  • 30
    I started a PhD in Statistics 9 months ago and to be honest Gelman's BDA is still above me, so I wouldn't call it an introductory text! – Sean Jun 28 '12 at 21:37
  • 4
    -1, because according to multiple comments and other answers, this isn't introductory. – naught101 Jul 24 '12 at 5:48
  • 6
    @naught101 so you downvote without knowing the book? – conjectures Jan 28 '15 at 12:25
  • 4
    The first four or five chapters are truly introductory! so belongs here. – kjetil b halvorsen Jan 28 '15 at 17:50

Another vote for Gelman et al., but a close second for me -- being of the learn-by-doing persuasion -- is Bayesian Computation with R.

  • 5
    Agree strongly. Both great books. Start with Bayesian Computation With R, then get Gelman et al. – PeterR Jul 20 '10 at 19:12

Statistical Rethinking, has been released just a few weeks ago and hence I am still reading it, but I think is a very nice and fresh addition to the really introductory books about Bayesian Statistics. The author uses a similar approach as the one used by John Kruschke in his puppy books; very verbose, detailed explanations, nice pedagogical examples, he also uses a computational rather than mathematical approach.

Youtube lectures and other material is also available from here.

Code ported to Python/PyMC3

  • 3
    +1 I'm listening through the lectures now. He's very entertaining, and has a good approach. The book is excellent and takes you from basics to hierarchical models. It only assumes that the reader is somewhat scientific, has a reasonable grasp of mathematics (not including calculus) and has heard some things about statistics. It's the book I wish I'd had. The order he presents things in, and his system of asides is brilliant. – Wayne Jan 20 '16 at 13:51
  • 1
    I hit a wall trying to work through Kruschke's book where he starting making some big leaps in logic that I just couldn't follow. Luckily, I came across Statistical Rethinking, which so far is the only book I've found that gives you a genuinely intuitive understanding of the topic. – Brideau Mar 27 '17 at 2:21

Sivia and Skilling, Data analysis: a Bayesian tutorial (2ed) 2006 246p 0198568320 books.goo:

Statistics lectures have been a source of much bewilderment and frustration for generations of students. This book attempts to remedy the situation by expounding a logical and unified approach to the whole subject of data analysis. This text is intended as a tutorial guide for senior undergraduates and research students in science and engineering ...

I don't know the other recommendations though.

  • 3
    This book it excellent. It's short and practical. – John Salvatier Nov 19 '10 at 1:48
  • 2
    I think this is a much better introductory text than Gelman. – Sean Jun 28 '12 at 21:38

For an introduction, I would recommend Probabilistic Programming & Bayesian Methods for Hackers by Cam Davidson-Pilon, freely available online.

From its description:

An intro to Bayesian methods and probabilistic programming from a computation/understanding-first, mathematics-second point of view.

It's highly visual, cuts straight to the value and backfills gritty details later, has lots of examples, has interactive code (in IPython Notebook).

  • 4
    I thought this online book was hard to follow / poorly written. – captain_ahab Apr 11 '15 at 22:37
  • 2
    I think the book is fine. – SmallChess Feb 9 '17 at 12:17
  • 1
    I think this book is a fantastic intro for programmers to have a great first experience with bayesian stats – SARose Apr 21 '17 at 4:20

I am an electrical engineer and not a statistician. I spent a lot of time to go through Gelman but I don't think one can refer to Gelman as introductory at all. My bayesian-guru professor from Carnegie Mellon agrees with me on this. having the minimum knowledge of statistics and R and Bugs(as the easy way to DO something with Bayesian stat) Doing Bayesian Data Analysis: A Tutorial with R and BUGS is an amazing start. You can compare all offered books easily by their book cover!

5 years later update: I want to add that perhaps one other major way of learning in a fast way(40 mins) is to go through the documentation of a Bayesian Net GUI based tool such as Netica2. It starts with basics, walks you through the steps of building a net based on a situation and data, and how to run your own questions back and forth to "get it!".

  • 1
    This is a duplicate of @rosser's answer above. As a truly introductory book, I've +1'd each. – Wayne Mar 12 '12 at 21:09

Its focus isn't strictly on Bayesian statistics, so it lacks some methodology, but David MacKay's Information Theory, Inference, and Learning Algorithms made me intuitively grasp Bayesian statistics better than others - most do the how quite nicely, but I felt MacKay explained why better.

  • 5
    And it is available for free download at the authors page: inference.phy.cam.ac.uk/mackay/itila/book.html – PeterR Sep 8 '10 at 12:37
  • 5
    Like Sivia, this is very nice if you have some physics background and can be rough if not. Not a good guide to any kind of applied social statistics (for that use Gelman and Hill, or Gelman et al. above) but really great for prompting you to really think about the core issues. – conjugateprior Dec 6 '10 at 21:12

The Gelman books are all excellent but not necessarily introductory in that they assume that you know some statistics already. Therefore they are an introduction to the Bayesian way of doing statistics rather than to statistics in general. I would still give them the thumbs up, however.

As an introductory statistics/econometrics book which takes a Bayesian perspective, I would recommend Gary Koop's Bayesian Econometrics.

I thoroughly recommend the entertaining polemic "Probability Theory: The Logic of Science" by E.T. Jaynes.

This is an introductory text in the sense of not requiring (and in fact preferring) no previous knowledge of statistics, but it does eventually employ fairly sophisticated mathematics. Compared to most of the other answers provided, this book is not nearly as practical or easy to digest, rather it provides the philosophical bedrock to why you would want to employ Bayesian methods, and why not to use frequentist approaches. It is introductory in a historical and philosophical, but not pedagogical way.

  • 7
    This is a brilliant book about Bayesian thinking rather than applying Bayesian methods. I think this is a good companion text to something which goes more into how do Bayesian computations. – probabilityislogic Jul 24 '12 at 12:29
  • 3
    That's a good way of putting it. I think Sivia and Skilling is an ideal companion text for introducing the methods in practice (which has already been suggested in another answer). – Bogdanovist Jul 24 '12 at 22:22
  • 1
    Entertaining and polemic and original, for sure, but definitely not an introductory book. – Xi'an Apr 27 at 11:05

"Bayesian Core: A Practical Approach to Computational Bayesian Statistics" by Marin and Robert, Springer-Verlag (2007)

  • 5
    @Xi'an and gappy, please explain why this book can be recommended. For whom is it suitable? In what sense is it "best"? – whuber Jan 25 '12 at 15:34
  • 3
    I do not want to fall into self-promotion. Bayesian Core is a self-contained entry to Bayesian inference for the most common models and to computational methods (R codes provided). It requires some background in probability theory that may be too much for some readers... (It works well with our 4th and 5th year students in France.) – Xi'an Jan 26 '12 at 14:48

My favourite first undergraduate text for bayesian statistics is by Bolstad, Introduction to Bayesian Statistics. If you're looking for something graduate level, this will be too elementary, but for someone who is new to statistics this is ideal.

I have read some parts of A First Course in Bayesian Statistical Methods by Peter Hoff, and I found it easy to follow. (Example R-code is provided throughout the text)

If you're looking for an elementary text, i.e. one that doesn't have a calculus prerequisite, there's Don Berry's Statistics: A Bayesian Perspective.

Take a look at "The Bayesian Choice". It has the full package: foundations, applications and computation. Clearly written.

  • Would not only be a „Bayesian“ but rather a „great choice“, if the solution manual were vailable for self-study. It seems this is intended for university use only... – gwr Nov 10 '17 at 23:44

I've at least glanced at most of these on this list and none are as good as the new Bayesian Ideas and Data Analysis in my opinion.

Edit: It is easy to immediately begin doing Bayesian analysis while reading this book. Not just model the mean from a Normal distribution with known variance, but actual data analysis after the first couple of chapters. All code examples and data are on the book's website. Covers a decent amount of theory but the focus is applications. Lots of examples over a wide range of models. Nice chapter on Bayesian Nonparametrics. Winbugs, R, and SAS examples. I prefer it over Doing Bayesian Data Analysis (I have both). Most of the books on here (Gelman, Robert, ...) are not introductory in my opinion and unless you have someone to talk to you will probably be left with more questions then answers. Albert's book does not cover enough material to feel comfortable analyzing data different from what is presented in the book (again my opinion).

  • 2
    "Good" in what sense? – whuber Jan 25 '12 at 15:34
  • Good point. Good as in best introductory Bayesian textbook. I believe it to be 'better' than Bayesian Data Analysis with R by Albert and I found Bayesian Data Analysis by Gelman et al. to not suffice as an introduction. After learning some Bayesian material however, it is a good reference. – Glen Apr 9 '12 at 14:18

I found an excellent introduction in Gelman and Hill (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models. (Other comments mention it, but it deserves to get upvoted on its own.)

I don't know why nobody has mentioned the very introductory book on Bayesian:

enter image description here

There's a free PDF version for the book. The book offers enough material for anyone who has very little experience in bayesian. It introduces the concept of prior distribution, posterior distribution, beta distribution etc.

Give it a go, it's free.

http://greenteapress.com/thinkbayes/

I simply must to include MCMC in Practice. It provides an excellent introduction to MCMC, perhaps not as general as other books mentioned, but excellent for gaining insight and intuition. I would recommend reading it after (or in parallel with) Bayesian Computation with R.

  • Mcmc should not be the focus of an introduction to bayesian statistics, in my opinion. I think rejection sampling is more appealling as a way to understand how bayesian learning works. Also, least squares is bayesian (as is maximum likehood), so it too represents a gentler introduction to bayesian statistics, compared to mcmc. – probabilityislogic Jan 23 '12 at 9:59
  • 2
    My view is that mcmc should be avoided and used as a last resort - it simply takes too long in most cases (though i deal with big data sets where everything is basically mle). mcmc is a "sledge hammer" to some degree. Also mcmc is an algorithm for numerical integration. Nothing more, nothing less. It should receive the same introductory treatment as other algorithms, such as the laplace method and quadratre. Otherwise people will develop a narrow view of what "bayesian statistics" is. – probabilityislogic Jan 23 '12 at 12:13

If you happen to come from the physical sciencies (physics/astronomy) I would recommend you Bayesian Logical Data Analysis for the Physical Sciences: A Comparative Approach with Mathematica® Support by Gregory (2006).

Although the "with Mathematica® Support" part of the title is there only for commercial issues (the usages of Mathematica code are very poor), the good thing about this book is that it is truly an introduction to the subject of probabilities and statistics. It even has some chapters on frequentist statistics. However, once you give it a shot, go for the book of Gelman et. al that a lot of people recommended you. Most of the material in the book of Gregory is taken lightly (if not, it wouldn't be an introduction): Gelman's book has been a truly re-awakening from Gregory's for me.

  • Phil Gregory's book is a very nice introduction indeed, somewhat like Bolstad's introduction for people with an advanced mathematical background. There are more resources to be found on Phil Gregory's website and there is also a supplement which addresses hierarchical models and missing data treatment. – gwr Nov 17 '15 at 11:27

I read:

Gelman et al (2013). Bayesian Data Analysis. CRC Press LLC. 3rd ed.

Hoff, Peter D (2009). A First Course in Bayesian Statistical Methods. Springer Texts in Statistics.

Kruschke, Doing Bayesian Data Analysis: A Tutorial with R and Bugs, 2011. Academic Press / Elsevier.

and I think that the better one to start with is Kruschke's book. It's perfect for a first approach to Bayesian thinking: concepts are explained very clearly, there is not too much mathematics, and there are lots of nice examples!

Gelman et al. is a great book, but it is more advanced and I suggest to read it after the Kruschke's one.

Conversely, I did not like Hoff's book because it is an introductory book, but concepts (and Bayesian thinking) are not explained in a clear way. I suggest to pass over.

If I had to choose a single text for a beginner, it would be

              Sivia DS and Skilling J (2006) book (see below). 

Of all the books listed below it strives hardest to give an intuitive grasp of the essential ideas, but it still requires some mathematical sophistication from page 1.

Below is a list of Further Readings from my book, with comments on each publication.

Bernardo, JM and Smith, A, (2000) 4 . Bayesian Theory A rigorous account of Bayesian methods, with many real-world examples.

Bishop, C (2006) 5 . Pattern Recognition and Machine Learning. As the title suggests, this is mainly about machine learning, but it provides a lucid and comprehensive account of Bayesian methods.

Cowan G (1998) 6 . Statistical Data Analysis. An excellent non-Bayesian introduction to statistical analysis.

Dienes, Z (2008) 8 . Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Provides tutorial material on Bayes’ rule and a lucid analysis of the distinction between Bayesian and frequentist statistics.

Gelman A, Carlin J, Stern H, and Rubin D. (2003) 14 . Bayesian Data Analysis. A rigorous and comprehensive account of Bayesian analysis, with many real-world examples.

Jaynes E and Bretthorst G (2003) 18 . Probability Theory: The Logic of Science. The modern classic of Bayesian analysis. It is comprehensive and wise. Its discursive style makes it long (600 pages) but never dull,and it is packed ful l of insights.

Khan, S, 2012, Introduction to Bayes’ Theorem. Salman Khan’s online mathematics videos make a good introduction to various topics, including Bayes’ rule.

Lee PM (2004) 27 . Bayesian Statistics: An Introduction. A rigorous and comprehensive text with a strident Bayesian style.

MacKay DJC (2003) 28 . Information theory, inference, and learning algorithms. The modern classic on information theory. A very readable text that roams far and wide over many topics, almost all of which make use of Bayes’ rule.

Migon, HS and Gamerman, D (1999) 30. Statistical Inference: An Integrated Approach. A straightforward (and clearly laid out) account of inference, which compares Bayesian and non-Bayesian approaches. Despite being fairly advanced, the writing style is tutorial in nature.

Pierce JR (1980) 34 2nd Edition. An introduction to information theory: symbols, signals and noise. Pierce writes with an informal, tutorial style of writing, but does not flinch from presenting the fundamental theorems of information theory.

Reza, FM (1961) 35 . An introduction to information theory. A more comprehensive and mathematical ly rigorous book than the Pierce book above, and should ideally be read only after first reading Pierce’s more informal text.

Sivia DS and Skilling J (2006) 38 . Data Analysis: A Bayesian Tutorial. This is an excellent tutorial style introduction to Bayesian methods.

Spiegelhalter, D and Rice, K (2009) 36 . Bayesian statistics. Scholarpedia, 4(8):5230. http://www.scholarpedia.org/article/Bayesian_statistics A reliable and comprehensive summary of the current status of Bayesian statistics.

And, here is my book, published June 2013.

Bayes' Rule: A Tutorial Introduction to Bayesian Analysis, Dr James V Stone, ISBN 978-0956372840

Chapter 1 can be downloaded from: http://jim-stone.staff.shef.ac.uk/BookBayes2012/BayesRuleBookMain.html

Description: Discovered by an 18th century mathematician and preacher, Bayes' rule is a cornerstone of modern probability theory. In this richly illustrated book, a range of accessible examples are used to show how Bayes' rule is actually a natural consequence of commonsense reasoning. Bayes' rule is derived using intuitive graphical representations of probability, and Bayesian analysis is applied to parameter estimation using the MatLab programs provided. The tutorial style of writing, combined with a comprehensive glossary, makes this an ideal primer for the novice who wishes to become familiar with the basic principles of Bayesian analysis.

enter image description here

Not strictly Bayesian Statistics as such, but I can strongly recommend "A First Course on Machine Learning" by Rogers and Girolami, which is essentially an introduction to Bayesian approaches to machine learning. Its very well structured and clear and aimed at students without a strong mathematical background. This means it is a pretty good first introduction to Bayesian ideas. There is also MATLAB/OCTAVE code which is a nice feature.

Bayesian Statistics for Social Scientists. Phillips, Lawrence D. (1973), Thomas Crowell & Co. It's very clear, very accessible, assumes no statistics knowledge, and, unlike Bolstad which I found dry, has some personality.

This book suggests it is aimed at entry level undergraduate level

Biostatistics: A Bayesian Introduction. By George G Woodsworth.

Published by John Wiley & Sons

Gelman et al. is well-regarded but explictly intended for a graduate course. If you don't have substantial prior coursework in statistics, it is largely a waste.

  • 5
    +0, this would serve better as a comment to @Shane's answer. You might edit your answer to recommend a book that would better sever the stated purpose for those without "substantial prior coursework in statistics". – gung Jan 23 '12 at 3:45

Gill, J. (2014). Bayesian Methods: A Social and Behavioral Sciences Approach. 3rd edition.

Written by a political science professor, with social scientists as the target audience in mind. R code is provided.

http://www.amazon.com/Bayesian-Methods-Behavioral-Sciences-Statistics/dp/1439862486/ enter image description here

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.