Which is the best introductory textbook for Bayesian statistics?
One book per answer, please.
$\begingroup$
$\endgroup$
3
-
43$\begingroup$ In the replies, please explain why you are recommending a book as "the best." $\endgroup$– whuber ♦Commented Jan 25, 2012 at 15:33
-
4$\begingroup$ How can there be more than one answer to a question question phrased like this? $\endgroup$– naught101Commented Jul 24, 2012 at 5:44
-
12$\begingroup$ This is an old thread now, but I came back to +1 a new book "Statistical Rethinking. And in looking the higher-ranking answers in the thread, I think a key distinction hasn't been made: "introductory" for whom? A first course in statistics (that happens to have a Bayesian approach)? An introduction to Bayesian methods for someone with basic undergraduate (non-Bayesian) statistics classes? Or an introduction to Bayesian statistics for a practitioner of non-Bayesian statistics who has finally been persuaded that this Bayesian thing isn't a fad? Very different introductions. $\endgroup$– WayneCommented Jan 20, 2016 at 13:54
Add a comment
|
38 Answers
$\begingroup$
$\endgroup$
7
John Kruschke released a book in mid 2011 called Doing Bayesian Data Analysis: A Tutorial with R and BUGS. (A second edition was released in Nov 2014: Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan.) It is truly introductory. If you want to walk from frequentist stats into Bayes though, especially with multilevel modelling, I recommend Gelman and Hill.
John Kruschke also has a website for the book that has all the examples in the book in BUGS and JAGS. His blog on Bayesian statistics also links in with the book.
-
$\begingroup$ @Amir's suggestion is a duplicate of this. (The full title of the book is "Doing Bayesian Data Analysis: A Tutorial with R and BUGS".) As a truly introductory book, I've +1'd each. $\endgroup$– WayneCommented Mar 12, 2012 at 21:10
-
$\begingroup$ updated the title and added a couple of related links. $\endgroup$ Commented Mar 21, 2012 at 22:49
-
5$\begingroup$ I also vote for Kruschke's book. I've browsed most of the books listed in the answers and this is the one I found most clear. IMO, it is the most clear stats book I have read. It helps a lot that R code is available to match formulas with code. The author starts with very simple examples and builds on them. Very little background is needed. All reviews on Amazon are highly favorable. Hoff's book is my second favorite. $\endgroup$– juliethCommented Jul 29, 2012 at 13:02
-
$\begingroup$ Haha, I like the book cover: "Why the happy puppies? (as if happy puppies needed justification!)" $\endgroup$– ZhubarbCommented Oct 25, 2013 at 9:21
-
$\begingroup$ My vote also goes to Kruschke's 2010 book. In trying to learn Bayesian statistics, I tried several of them, and this one hit the mark. Hard. $\endgroup$ Commented Jan 24, 2014 at 0:49
$\begingroup$
$\endgroup$
10
My favorite is "Bayesian Data Analysis" by Gelman, et al. (The pdf version is legally free since April 2020!)
-
34$\begingroup$ This is an introductory book for people who have a decent amount of statistical background already. $\endgroup$ Commented Nov 20, 2010 at 23:41
-
52$\begingroup$ I started a PhD in Statistics 9 months ago and to be honest Gelman's BDA is still above me, so I wouldn't call it an introductory text! $\endgroup$– SeanCommented Jun 28, 2012 at 21:37
-
2$\begingroup$ @Shane, can you please to this answer explaining why it's your favourite? $\endgroup$ Commented Jul 24, 2012 at 5:45
-
11$\begingroup$ @naught101 so you downvote without knowing the book? $\endgroup$ Commented Jan 28, 2015 at 12:25
-
5$\begingroup$ The first four or five chapters are truly introductory! so belongs here. $\endgroup$ Commented Jan 28, 2015 at 17:50
$\begingroup$
$\endgroup$
7
Statistical Rethinking, has been released just a few weeks ago and hence I am still reading it, but I think is a very nice and fresh addition to the really introductory books about Bayesian Statistics. The author uses a similar approach as the one used by John Kruschke in his puppy books; very verbose, detailed explanations, nice pedagogical examples, he also uses a computational rather than mathematical approach.
Youtube lectures and other material is also available from here.
-
4$\begingroup$ +1 I'm listening through the lectures now. He's very entertaining, and has a good approach. The book is excellent and takes you from basics to hierarchical models. It only assumes that the reader is somewhat scientific, has a reasonable grasp of mathematics (not including calculus) and has heard some things about statistics. It's the book I wish I'd had. The order he presents things in, and his system of asides is brilliant. $\endgroup$– WayneCommented Jan 20, 2016 at 13:51
-
2$\begingroup$ I hit a wall trying to work through Kruschke's book where he starting making some big leaps in logic that I just couldn't follow. Luckily, I came across Statistical Rethinking, which so far is the only book I've found that gives you a genuinely intuitive understanding of the topic. $\endgroup$– BrideauCommented Mar 27, 2017 at 2:21
-
3$\begingroup$ After going through the thread, I tried reading the first chapter of this book, and I found it very difficult as a non-native English speaker and as a non-scientist. First I had to go through the words like epistemology, idiosyncratic, then there are long sentences, which I had to read twice/thrice to understand what tehy means literally (forget about the conclusion of those sentences). Then the very first example is about natural evolution, which sounded Greek to me: number of sites, number of alleles, neutrality. The book could be easy for a lot, but could be difficult for many $\endgroup$ Commented Sep 30, 2018 at 12:27
-
$\begingroup$ A second edition of this (superb) book will be coming out in March 2020. $\endgroup$– eipi10Commented Jan 3, 2020 at 5:16
-
2$\begingroup$ Glad to see there's a link to the Python notebooks! I don't use R and would prefer to use my time to learn Bayesian inference instead of coding in R (for now!). $\endgroup$– ireneCommented Jul 29, 2021 at 3:27
$\begingroup$
$\endgroup$
3
Sivia and Skilling, Data analysis: a Bayesian tutorial (2ed) 2006 246p 0198568320 books.goo:
Statistics lectures have been a source of much bewilderment and frustration for generations of students. This book attempts to remedy the situation by expounding a logical and unified approach to the whole subject of data analysis. This text is intended as a tutorial guide for senior undergraduates and research students in science and engineering ...
I don't know the other recommendations though.
-
4$\begingroup$ This book it excellent. It's short and practical. $\endgroup$ Commented Nov 19, 2010 at 1:48
-
3$\begingroup$ I think this is a much better introductory text than Gelman. $\endgroup$– SeanCommented Jun 28, 2012 at 21:38
-
$\begingroup$
$\endgroup$
2
Another vote for Gelman et al., but a close second for me -- being of the learn-by-doing persuasion -- is Jim Albert's "Bayesian Computation with R".
-
6$\begingroup$ Agree strongly. Both great books. Start with Bayesian Computation With R, then get Gelman et al. $\endgroup$– PeterRCommented Jul 20, 2010 at 19:12
-
$\begingroup$ Link is dead. Maybe this one? bayesball.github.io/bcwr/index.html $\endgroup$ Commented Jan 25, 2021 at 23:43
$\begingroup$
$\endgroup$
3
For an introduction, I would recommend Probabilistic Programming & Bayesian Methods for Hackers by Cam Davidson-Pilon, freely available online.
From its description:
An intro to Bayesian methods and probabilistic programming from a computation/understanding-first, mathematics-second point of view.
It's highly visual, cuts straight to the value and backfills gritty details later, has lots of examples, has interactive code (in IPython Notebook).
-
4$\begingroup$ I thought this online book was hard to follow / poorly written. $\endgroup$ Commented Apr 11, 2015 at 22:37
-
3
-
3$\begingroup$ I think this book is a fantastic intro for programmers to have a great first experience with bayesian stats $\endgroup$– SARoseCommented Apr 21, 2017 at 4:20
$\begingroup$
$\endgroup$
3
I thoroughly recommend the entertaining polemic "Probability Theory: The Logic of Science" by E.T. Jaynes.
This is an introductory text in the sense of not requiring (and in fact preferring) no previous knowledge of statistics, but it does eventually employ fairly sophisticated mathematics. Compared to most of the other answers provided, this book is not nearly as practical or easy to digest, rather it provides the philosophical bedrock to why you would want to employ Bayesian methods, and why not to use frequentist approaches. It is introductory in a historical and philosophical, but not pedagogical way.
-
9$\begingroup$ This is a brilliant book about Bayesian thinking rather than applying Bayesian methods. I think this is a good companion text to something which goes more into how do Bayesian computations. $\endgroup$ Commented Jul 24, 2012 at 12:29
-
5$\begingroup$ That's a good way of putting it. I think Sivia and Skilling is an ideal companion text for introducing the methods in practice (which has already been suggested in another answer). $\endgroup$ Commented Jul 24, 2012 at 22:22
-
5$\begingroup$ Entertaining and polemic and original, for sure, but definitely not an introductory book. $\endgroup$– Xi'anCommented Apr 27, 2018 at 11:05
$\begingroup$
$\endgroup$
2
Its focus isn't strictly on Bayesian statistics, so it lacks some methodology, but David MacKay's Information Theory, Inference, and Learning Algorithms made me intuitively grasp Bayesian statistics better than others - most do the how quite nicely, but I felt MacKay explained why better.
-
8$\begingroup$ And it is available for free download at the authors page: inference.phy.cam.ac.uk/mackay/itila/book.html $\endgroup$– PeterRCommented Sep 8, 2010 at 12:37
-
7$\begingroup$ Like Sivia, this is very nice if you have some physics background and can be rough if not. Not a good guide to any kind of applied social statistics (for that use Gelman and Hill, or Gelman et al. above) but really great for prompting you to really think about the core issues. $\endgroup$ Commented Dec 6, 2010 at 21:12
$\begingroup$
$\endgroup$
1
I am an electrical engineer and not a statistician. I spent a lot of time to go through Gelman but I don't think one can refer to Gelman as introductory at all. My bayesian-guru professor from Carnegie Mellon agrees with me on this. having the minimum knowledge of statistics and R and Bugs(as the easy way to DO something with Bayesian stat) Doing Bayesian Data Analysis: A Tutorial with R and BUGS is an amazing start. You can compare all offered books easily by their book cover!
5 years later update: I want to add that perhaps one other major way of learning in a fast way(40 mins) is to go through the documentation of a Bayesian Net GUI based tool such as Netica2. It starts with basics, walks you through the steps of building a net based on a situation and data, and how to run your own questions back and forth to "get it!".
-
1$\begingroup$ This is a duplicate of @rosser's answer above. As a truly introductory book, I've +1'd each. $\endgroup$– WayneCommented Mar 12, 2012 at 21:09
$\begingroup$
$\endgroup$
The Gelman books are all excellent but not necessarily introductory in that they assume that you know some statistics already. Therefore they are an introduction to the Bayesian way of doing statistics rather than to statistics in general. I would still give them the thumbs up, however.
As an introductory statistics/econometrics book which takes a Bayesian perspective, I would recommend Gary Koop's Bayesian Econometrics.
$\begingroup$
$\endgroup$
1
I don't know why nobody has mentioned the very introductory book on Bayesian:
There's a free PDF version for the book. The book offers enough material for anyone who has very little experience in bayesian. It introduces the concept of prior distribution, posterior distribution, beta distribution etc.
Give it a go, it's free.
-
$\begingroup$ The 2nd Edition is out now. Free online with runnable Python notebooks: allendowney.github.io/ThinkBayes2 $\endgroup$– jeffhaleCommented Aug 27, 2021 at 13:45
$\begingroup$
$\endgroup$
2
"Bayesian Core: A Practical Approach to Computational Bayesian Statistics" by Marin and Robert, Springer-Verlag (2007).
"Why?": the author explain the why of the bayesian choice and the how very well. It's a practical book, but written by one of the finest bayesian thinkers alive. It's not exhaustive. Other books have that objective. It picks up a few topics that are relevant, useful, and illuminating the foundations.
About "choice": if you really want to delve into bayesian foundation, Xi'an' "The Bayesian Choice" is clear, deep, essential.
-
7$\begingroup$ @Xi'an and gappy, please explain why this book can be recommended. For whom is it suitable? In what sense is it "best"? $\endgroup$– whuber ♦Commented Jan 25, 2012 at 15:34
-
4$\begingroup$ I do not want to fall into self-promotion. Bayesian Core is a self-contained entry to Bayesian inference for the most common models and to computational methods (R codes provided). It requires some background in probability theory that may be too much for some readers... (It works well with our 4th and 5th year students in France.) $\endgroup$– Xi'anCommented Jan 26, 2012 at 14:48
$\begingroup$
$\endgroup$
0
My favourite first undergraduate text for bayesian statistics is by Bolstad, Introduction to Bayesian Statistics. If you're looking for something graduate level, this will be too elementary, but for someone who is new to statistics this is ideal.
$\begingroup$
$\endgroup$
I have read some parts of A First Course in Bayesian Statistical Methods by Peter Hoff, and I found it easy to follow. (Example R-code is provided throughout the text)
$\begingroup$
$\endgroup$
Coming from non-statistical background I found Introduction to Applied Bayesian Statistics and Estimation for Social Scientists quite informative and easy to follow.
$\begingroup$
$\endgroup$
If you're looking for an elementary text, i.e. one that doesn't have a calculus prerequisite, there's Don Berry's Statistics: A Bayesian Perspective.
$\begingroup$
$\endgroup$
I found an excellent introduction in Gelman and Hill (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models. (Other comments mention it, but it deserves to get upvoted on its own.)
$\begingroup$
$\endgroup$
I quite like Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference by Gamerman and Lopes.
$\begingroup$
$\endgroup$
1
Take a look at "The Bayesian Choice". It has the full package: foundations, applications and computation. Clearly written.
-
2$\begingroup$ Would not only be a „Bayesian“ but rather a „great choice“, if the solution manual were vailable for self-study. It seems this is intended for university use only... $\endgroup$– gwrCommented Nov 10, 2017 at 23:44
$\begingroup$
$\endgroup$
2
I've at least glanced at most of these on this list and none are as good as the new Bayesian Ideas and Data Analysis in my opinion.
Edit: It is easy to immediately begin doing Bayesian analysis while reading this book. Not just model the mean from a Normal distribution with known variance, but actual data analysis after the first couple of chapters. All code examples and data are on the book's website. Covers a decent amount of theory but the focus is applications. Lots of examples over a wide range of models. Nice chapter on Bayesian Nonparametrics. Winbugs, R, and SAS examples. I prefer it over Doing Bayesian Data Analysis (I have both). Most of the books on here (Gelman, Robert, ...) are not introductory in my opinion and unless you have someone to talk to you will probably be left with more questions then answers. Albert's book does not cover enough material to feel comfortable analyzing data different from what is presented in the book (again my opinion).
-
2
-
$\begingroup$ Good point. Good as in best introductory Bayesian textbook. I believe it to be 'better' than Bayesian Data Analysis with R by Albert and I found Bayesian Data Analysis by Gelman et al. to not suffice as an introduction. After learning some Bayesian material however, it is a good reference. $\endgroup$– GlenCommented Apr 9, 2012 at 14:18
$\begingroup$
$\endgroup$
If I had to choose a single text for a beginner, it would be
Sivia DS and Skilling J (2006) book (see below).
Of all the books listed below it strives hardest to give an intuitive grasp of the essential ideas, but it still requires some mathematical sophistication from page 1.
Below is a list of Further Readings from my book, with comments on each publication.
Bernardo, JM and Smith, A, (2000) 4 . Bayesian Theory A rigorous account of Bayesian methods, with many real-world examples.
Bishop, C (2006) 5 . Pattern Recognition and Machine Learning. As the title suggests, this is mainly about machine learning, but it provides a lucid and comprehensive account of Bayesian methods.
Cowan G (1998) 6 . Statistical Data Analysis. An excellent non-Bayesian introduction to statistical analysis.
Dienes, Z (2008) 8 . Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Provides tutorial material on Bayes’ rule and a lucid analysis of the distinction between Bayesian and frequentist statistics.
Gelman A, Carlin J, Stern H, and Rubin D. (2003) 14 . Bayesian Data Analysis. A rigorous and comprehensive account of Bayesian analysis, with many real-world examples.
Jaynes E and Bretthorst G (2003) 18 . Probability Theory: The Logic of Science. The modern classic of Bayesian analysis. It is comprehensive and wise. Its discursive style makes it long (600 pages) but never dull,and it is packed ful l of insights.
Khan, S, 2012, Introduction to Bayes’ Theorem. Salman Khan’s online mathematics videos make a good introduction to various topics, including Bayes’ rule.
Lee PM (2004) 27 . Bayesian Statistics: An Introduction. A rigorous and comprehensive text with a strident Bayesian style.
MacKay DJC (2003) 28 . Information theory, inference, and learning algorithms. The modern classic on information theory. A very readable text that roams far and wide over many topics, almost all of which make use of Bayes’ rule.
Migon, HS and Gamerman, D (1999) 30. Statistical Inference: An Integrated Approach. A straightforward (and clearly laid out) account of inference, which compares Bayesian and non-Bayesian approaches. Despite being fairly advanced, the writing style is tutorial in nature.
Pierce JR (1980) 34 2nd Edition. An introduction to information theory: symbols, signals and noise. Pierce writes with an informal, tutorial style of writing, but does not flinch from presenting the fundamental theorems of information theory.
Reza, FM (1961) 35 . An introduction to information theory. A more comprehensive and mathematical ly rigorous book than the Pierce book above, and should ideally be read only after first reading Pierce’s more informal text.
Sivia DS and Skilling J (2006) 38 . Data Analysis: A Bayesian Tutorial. This is an excellent tutorial style introduction to Bayesian methods.
Spiegelhalter, D and Rice, K (2009) 36 . Bayesian statistics. Scholarpedia, 4(8):5230. http://www.scholarpedia.org/article/Bayesian_statistics A reliable and comprehensive summary of the current status of Bayesian statistics.
And, here is my book, published June 2013.
Bayes' Rule: A Tutorial Introduction to Bayesian Analysis, Dr James V Stone, ISBN 978-0956372840
Chapter 1 can be downloaded from: http://jim-stone.staff.shef.ac.uk/BookBayes2012/BayesRuleBookMain.html
Description: Discovered by an 18th century mathematician and preacher, Bayes' rule is a cornerstone of modern probability theory. In this richly illustrated book, a range of accessible examples are used to show how Bayes' rule is actually a natural consequence of commonsense reasoning. Bayes' rule is derived using intuitive graphical representations of probability, and Bayesian analysis is applied to parameter estimation using the MatLab programs provided. The tutorial style of writing, combined with a comprehensive glossary, makes this an ideal primer for the novice who wishes to become familiar with the basic principles of Bayesian analysis.
$\begingroup$
$\endgroup$
0
For complete beginners, try William Briggs Breaking the Law of Averages: Real-Life Probability and Statistics in Plain English
$\begingroup$
$\endgroup$
2
I simply must to include MCMC in Practice. It provides an excellent introduction to MCMC, perhaps not as general as other books mentioned, but excellent for gaining insight and intuition. I would recommend reading it after (or in parallel with) Bayesian Computation with R.
-
$\begingroup$ Mcmc should not be the focus of an introduction to bayesian statistics, in my opinion. I think rejection sampling is more appealling as a way to understand how bayesian learning works. Also, least squares is bayesian (as is maximum likehood), so it too represents a gentler introduction to bayesian statistics, compared to mcmc. $\endgroup$ Commented Jan 23, 2012 at 9:59
-
2$\begingroup$ My view is that mcmc should be avoided and used as a last resort - it simply takes too long in most cases (though i deal with big data sets where everything is basically mle). mcmc is a "sledge hammer" to some degree. Also mcmc is an algorithm for numerical integration. Nothing more, nothing less. It should receive the same introductory treatment as other algorithms, such as the laplace method and quadratre. Otherwise people will develop a narrow view of what "bayesian statistics" is. $\endgroup$ Commented Jan 23, 2012 at 12:13
$\begingroup$
$\endgroup$
1
If you happen to come from the physical sciencies (physics/astronomy) I would recommend you Bayesian Logical Data Analysis for the Physical Sciences: A Comparative Approach with Mathematica® Support by Gregory (2006).
Although the "with Mathematica® Support" part of the title is there only for commercial issues (the usages of Mathematica code are very poor), the good thing about this book is that it is truly an introduction to the subject of probabilities and statistics. It even has some chapters on frequentist statistics. However, once you give it a shot, go for the book of Gelman et. al that a lot of people recommended you. Most of the material in the book of Gregory is taken lightly (if not, it wouldn't be an introduction): Gelman's book has been a truly re-awakening from Gregory's for me.
-
$\begingroup$ Phil Gregory's book is a very nice introduction indeed, somewhat like Bolstad's introduction for people with an advanced mathematical background. There are more resources to be found on Phil Gregory's website and there is also a supplement which addresses hierarchical models and missing data treatment. $\endgroup$– gwrCommented Nov 17, 2015 at 11:27
$\begingroup$
$\endgroup$
I read:
Gelman et al (2013). Bayesian Data Analysis. CRC Press LLC. 3rd ed.
Hoff, Peter D (2009). A First Course in Bayesian Statistical Methods. Springer Texts in Statistics.
Kruschke, Doing Bayesian Data Analysis: A Tutorial with R and Bugs, 2011. Academic Press / Elsevier.
and I think that the better one to start with is Kruschke's book. It's perfect for a first approach to Bayesian thinking: concepts are explained very clearly, there is not too much mathematics, and there are lots of nice examples!
Gelman et al. is a great book, but it is more advanced and I suggest to read it after the Kruschke's one.
Conversely, I did not like Hoff's book because it is an introductory book, but concepts (and Bayesian thinking) are not explained in a clear way. I suggest to pass over.
$\begingroup$
$\endgroup$
Not strictly Bayesian Statistics as such, but I can strongly recommend "A First Course on Machine Learning" by Rogers and Girolami, which is essentially an introduction to Bayesian approaches to machine learning. Its very well structured and clear and aimed at students without a strong mathematical background. This means it is a pretty good first introduction to Bayesian ideas. There is also MATLAB/OCTAVE code which is a nice feature.
$\begingroup$
$\endgroup$
Bayesian Statistics for Social Scientists. Phillips, Lawrence D. (1973), Thomas Crowell & Co. It's very clear, very accessible, assumes no statistics knowledge, and, unlike Bolstad which I found dry, has some personality.
$\begingroup$
$\endgroup$
1
A good book from the basics to advanced, and which you can download, is
Andrew Gelman, John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Donald Rubin, Bayesian Data Analysis, http://www.stat.columbia.edu/~gelman/book/
You can also download the first two chapters of
Richard McElreath, A Bayesian Course with Examples in R and Stan, https://xcelab.net/rm/statistical-rethinking/
-
1$\begingroup$ I highly recommend Dr. McElreath's book if you don't have rigorous Statistical Inference training. $\endgroup$ Commented Sep 30, 2020 at 4:35
$\begingroup$
$\endgroup$
This book suggests it is aimed at entry level undergraduate level
Biostatistics: A Bayesian Introduction. By George G Woodsworth.
Published by John Wiley & Sons
$\begingroup$
$\endgroup$
Computational Bayesian Statistics by Turkman et. al. is a high-quality and all-inclusive introduction to Bayesian statistics and its computational aspects. It has the right mix of theory, model assessment and selection, and a dedicated chapter on software for Bayesian statistics (with code examples). It should serve nicely as a practical textbook for a first course in Bayesian methods.
