What is the best introductory Bayesian statistics textbook? Which is the best introductory textbook for Bayesian statistics?
One book per answer, please.
 A: John Kruschke released a book in mid 2011 called Doing Bayesian Data Analysis: A Tutorial with R and BUGS. (A second edition was released in Nov 2014: Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan.) It is truly introductory. If you want to walk from frequentist stats into Bayes though, especially with multilevel modelling, I recommend Gelman and Hill.
John Kruschke also has a website for the book that has all the examples in the book in BUGS and JAGS. His blog on Bayesian statistics also links in with the book.
A: I've at least glanced at most of these on this list and none are as good as the new Bayesian Ideas and Data Analysis in my opinion.
Edit: It is easy to immediately begin doing Bayesian analysis while reading this book.  Not just model the mean from a Normal distribution with known variance, but actual data analysis after the first couple of chapters.  All code examples and data are on the book's website.   Covers a decent amount of theory but the focus is applications.  Lots of examples over a wide range of models.  Nice chapter on Bayesian Nonparametrics.  Winbugs, R, and SAS examples.  I prefer it over Doing Bayesian Data Analysis (I have both).  Most of the books on here (Gelman, Robert, ...) are not introductory in my opinion and unless you have someone to talk to you will probably be left with more questions then answers.  Albert's book does not cover enough material to feel comfortable analyzing data different from what is presented in the book (again my opinion).  
A: Take a look at "The Bayesian Choice". It has the full package: foundations, applications and computation. Clearly written.
A: I quite like Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference by Gamerman and Lopes.
A: If I had to choose a single text for a beginner, it would be 
              Sivia DS and Skilling J (2006) book (see below). 

Of all the books listed below it strives hardest to give an intuitive grasp of the essential ideas, but it still requires some mathematical sophistication from page 1.
Below is a list of Further Readings from my book, with comments on each publication.
Bernardo, JM and Smith, A, (2000) 4 . Bayesian Theory A rigorous account of Bayesian methods, with many real-world examples. 
Bishop, C (2006) 5 . Pattern Recognition and Machine Learning. As the title suggests, this is mainly about machine learning, but it provides a lucid and comprehensive account of Bayesian methods. 
Cowan G (1998) 6 . Statistical Data Analysis. An excellent non-Bayesian introduction to statistical analysis. 
Dienes, Z (2008) 8 . Understanding Psychology as a Science: An Introduction to Scientiﬁc and Statistical Inference. Provides tutorial material on Bayes’ rule and a lucid analysis of the 
distinction between Bayesian and frequentist statistics. 
Gelman A, Carlin J, Stern H, and Rubin D. (2003) 14 . Bayesian Data Analysis. A rigorous and comprehensive account of Bayesian analysis, with many real-world examples. 
Jaynes E and Bretthorst G (2003) 18 . Probability Theory: The Logic of Science. The modern classic of Bayesian analysis. It is comprehensive and wise. Its discursive style makes it long (600 pages) but never dull,and it is packed ful l of insights. 
Khan, S, 2012, Introduction to Bayes’ Theorem. Salman Khan’s online mathematics videos make a good introduction to various topics, including Bayes’ rule. 
Lee PM (2004) 27 . Bayesian Statistics: An Introduction. A rigorous and comprehensive text with a strident Bayesian style. 
MacKay DJC (2003) 28 . Information theory, inference, and learning algorithms. The modern classic on information theory. A very readable text that roams far and wide over many topics, almost all of which make use of Bayes’ rule. 
Migon, HS and Gamerman, D (1999) 30. Statistical Inference: An Integrated Approach. A straightforward (and clearly laid out) account of inference, which compares Bayesian and non-Bayesian approaches. Despite being fairly advanced, the writing style is tutorial in nature. 
Pierce JR (1980) 34 2nd Edition. An introduction to information theory: symbols, signals and noise. Pierce writes with an informal, tutorial  style of writing, but does not ﬂinch from presenting the fundamental theorems of information theory. 
Reza, FM (1961) 35 . An introduction to information theory. A more comprehensive and mathematical ly rigorous book than the Pierce book above, and should ideally be read only after ﬁrst reading Pierce’s more informal text. 
Sivia DS and Skilling J (2006) 38 . Data Analysis: A Bayesian Tutorial. This is an excellent tutorial style introduction to Bayesian methods. 
Spiegelhalter, D and Rice, K (2009) 36 . Bayesian statistics. Scholarpedia, 4(8):5230. 
http://www.scholarpedia.org/article/Bayesian_statistics 
A reliable and comprehensive summary of the current status of Bayesian 
statistics. 
And, here is my book, published June 2013.
Bayes' Rule: A Tutorial Introduction to Bayesian Analysis,
Dr James V Stone, ISBN 978-0956372840
Chapter 1 can be downloaded from: 
http://jim-stone.staff.shef.ac.uk/BookBayes2012/BayesRuleBookMain.html
Description:
Discovered by an 18th century mathematician and preacher, Bayes' rule is a cornerstone of modern probability theory. In this richly illustrated book, a range of accessible examples are used to show how Bayes' rule is actually a natural consequence of commonsense reasoning. Bayes' rule is derived using intuitive graphical representations of probability, and Bayesian analysis is applied to parameter estimation using the MatLab programs provided. The tutorial style of writing, combined with a comprehensive glossary, makes this an ideal primer for the novice who wishes to become familiar with the basic principles of Bayesian analysis.

A: For complete beginners, try William Briggs Breaking the Law of Averages: Real-Life Probability and Statistics in Plain English
A: I simply must to include MCMC in Practice.  It provides an excellent introduction to MCMC, perhaps not as general as other books mentioned, but excellent for gaining insight and intuition.  I would recommend reading it after (or in parallel with) Bayesian Computation with R.
A: My favorite is "Bayesian Data Analysis" by Gelman, et al. (The pdf version is legally free since April 2020!)
A: If you happen to come from the physical sciencies (physics/astronomy) I would recommend you Bayesian Logical Data Analysis for the Physical Sciences: A Comparative Approach with Mathematica® Support by Gregory (2006).
Although the "with Mathematica® Support" part of the title is there only for commercial issues (the usages of Mathematica code are very poor), the good thing about this book is that it is truly an introduction to the subject of probabilities and statistics. It even has some chapters on frequentist statistics. However, once you give it a shot, go for the book of Gelman et. al that a lot of people recommended you. Most of the material in the book of Gregory is taken lightly (if not, it wouldn't be an introduction): Gelman's book has been a truly re-awakening from Gregory's for me.
A: I read:
Gelman et al (2013). Bayesian Data Analysis. CRC Press LLC. 3rd ed.
Hoff, Peter D (2009). A First Course in Bayesian Statistical Methods. Springer Texts in Statistics.
Kruschke, Doing Bayesian Data Analysis: A Tutorial with R and Bugs, 2011. Academic Press / Elsevier.
and I think that the better one to start with is Kruschke's book. It's perfect for a first approach to Bayesian thinking: concepts are explained very clearly, there is not too much mathematics, and there are lots of nice examples!
Gelman et al. is a great book, but it is more advanced and I suggest to read it after the Kruschke's one.
Conversely, I did not like Hoff's book because it is an introductory book, but concepts (and Bayesian thinking) are not explained in a clear way. I suggest to pass over. 
A: Statistical Rethinking, has been released just a few weeks ago and hence I am still reading it, but I think is a very nice and fresh addition to the really introductory books about Bayesian Statistics. The author uses a similar approach as the one used by John Kruschke in his puppy books; very verbose, detailed explanations, nice pedagogical examples, he also uses a computational rather than mathematical approach. 
Youtube lectures and other material is also available from here.
Code ported to Python/PyMC3
A: Not strictly Bayesian Statistics as such, but I can strongly recommend "A First Course on Machine Learning" by Rogers and Girolami, which is essentially an introduction to Bayesian approaches to machine learning.  Its very well structured and clear and aimed at students without a strong mathematical background.  This means it is a pretty good first introduction to Bayesian ideas.  There is also MATLAB/OCTAVE code which is a nice feature.
A: Bayesian Statistics for Social Scientists. Phillips, Lawrence D. (1973), Thomas Crowell & Co. It's very clear, very accessible, assumes no statistics knowledge, and, unlike Bolstad which I found dry, has some personality.
A: Sivia and Skilling, Data analysis: a Bayesian tutorial (2ed) 2006 246p 0198568320
books.goo:

Statistics lectures have been a source
  of much bewilderment and frustration
  for generations of students. This book
  attempts to remedy the situation by
  expounding a logical and unified
  approach to the whole subject of data
  analysis. This text is intended as a
  tutorial guide for senior
  undergraduates and research students
  in science and engineering ...

I don't know the other recommendations though.
A: Another vote for Gelman et al., but a close second for me -- being of the learn-by-doing persuasion -- is Jim Albert's "Bayesian Computation with R".
A: This book suggests it is aimed at entry level undergraduate level
Biostatistics: A Bayesian Introduction.  By George G Woodsworth.
Published by John Wiley & Sons
A: Computational Bayesian Statistics by Turkman et. al. is a high-quality and all-inclusive introduction to Bayesian statistics and its computational aspects. It has the right mix of theory, model assessment and selection, and a dedicated chapter on software for Bayesian statistics (with code examples). It should serve nicely as a practical textbook for a first course in Bayesian methods.

A: A good book from the basics to advanced, and which you can download, is
Andrew Gelman, John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Donald Rubin, Bayesian Data Analysis, http://www.stat.columbia.edu/~gelman/book/
You can also download the first two chapters of
Richard McElreath, A Bayesian Course with Examples in R and Stan, https://xcelab.net/rm/statistical-rethinking/
A: For an introduction, I would recommend Probabilistic Programming & Bayesian Methods for Hackers by Cam Davidson-Pilon, freely available online.
From its description:

An intro to Bayesian methods and probabilistic programming from a computation/understanding-first, mathematics-second point of view.

It's highly visual, cuts straight to the value and backfills gritty details later, has lots of examples, has interactive code (in IPython Notebook).
A: I thoroughly recommend the entertaining polemic "Probability Theory: The Logic of Science" by E.T. Jaynes. 
This is an introductory text in the sense of not requiring (and in fact preferring) no previous knowledge of statistics, but it does eventually employ fairly sophisticated mathematics. Compared to most of the other answers provided, this book is not nearly as practical or easy to digest, rather it provides the philosophical bedrock to why you would want to employ Bayesian methods, and why not to use frequentist approaches. It is introductory in a historical and philosophical, but not pedagogical way.   
A: Gill, J. (2014). Bayesian Methods: A Social and Behavioral Sciences Approach. 3rd edition.
Written by a political science professor, with social scientists as the target audience in mind. R code is provided.
http://www.amazon.com/Bayesian-Methods-Behavioral-Sciences-Statistics/dp/1439862486/

A: Since the type of beginner is not specified in the question, here is my advice for beginning statisticians:
Andrew B. Lawson and Emmanuel Lesaffre (2012): Bayesian Biostatistics
This book was used in the first year of our statistical science master and I found it relatively easy to understand for such a difficult subject. As with the majority of 'biostatistics' books, the examples are mainly clinical biology, but the methods are not restricted to those useful in clinical science. We had had about half a year of statistical education prior to this and besides Bayes theorem, Bayesian statistics hadn't been introduced yet.
What's also nice is that the entire 649 slides of accompanying presentations are available online.
A: This book hasn't been mentioned yet, as it's relatively new (2018):
A Student's Guide to Bayesian Statistics by Ben Lambert
In his videos, Lambert explains concepts in a very intuitive manner. This book does the same, explaining why the math works, using plenty of examples, and including problems to work at the end of each chapter.  He incorporates videos into the book, noting in the margin when a particular video complements the given concept being explained.  
It's somewhat in the same vein as Statistical Rethinking, but I found both the book and his videos to be more to the point, while still explaining the concepts well.  
I'd suggest watching a few videos to get a feel for his style.
Ben Lambert's YouTube channel
A: Its focus isn't strictly on Bayesian statistics, so it lacks some methodology, but David MacKay's Information Theory, Inference, and Learning Algorithms made me intuitively grasp Bayesian statistics better than others - most do the how quite nicely, but I felt MacKay explained why better.
A: I am an electrical engineer and not a statistician. I spent a lot of time to go through Gelman but I don't think one can refer to Gelman as introductory at all. My bayesian-guru professor from Carnegie Mellon agrees with me on this. having the minimum knowledge of statistics and R and Bugs(as the easy way to DO something with Bayesian stat)  Doing Bayesian Data Analysis: A Tutorial with R and BUGS  is an amazing start. You can compare all offered books easily by their book cover!
5 years later update: I want to add that perhaps one other major way of learning in a fast way(40 mins) is to go through the documentation of a Bayesian Net GUI based tool such as Netica2. It starts with basics, walks you through the steps of building a net based on a situation and data, and how to run your own questions back and forth to "get it!". 
A: The Gelman books are all excellent but not necessarily introductory in that they assume that you know some statistics already. Therefore they are an introduction to the Bayesian way of doing statistics rather than to statistics in general. I would still give them the thumbs up, however.
As an introductory statistics/econometrics book which takes a Bayesian perspective, I would recommend Gary Koop's Bayesian Econometrics.
A: "Bayesian Core: A Practical Approach to Computational Bayesian Statistics" by Marin and Robert, Springer-Verlag (2007).
"Why?": the author explain the why of the bayesian choice and the how very well. It's a practical book, but written by one of the finest bayesian thinkers alive. It's not exhaustive. Other books have that objective. It picks up a few topics that are relevant, useful, and illuminating the foundations.
About "choice": if you really want to delve into bayesian foundation, Xi'an' "The Bayesian Choice" is clear, deep,  essential. 
A: I don't know why nobody has mentioned the very introductory book on Bayesian:

There's a free PDF version for the book. The book offers enough material for anyone who has very little experience in bayesian. It introduces the concept of prior distribution, posterior distribution, beta distribution etc.
Give it a go, it's free.
http://greenteapress.com/thinkbayes/
A: My favourite first undergraduate text for bayesian statistics is by Bolstad, Introduction to Bayesian Statistics.  If you're looking for something graduate level, this will be too elementary, but for someone who is new to statistics this is ideal.
A: I have read some parts of A First Course in Bayesian Statistical Methods by Peter Hoff, and I found it easy to follow. (Example R-code is provided throughout the text)
A: Coming from non-statistical background I found Introduction to Applied Bayesian Statistics and Estimation for Social Scientists quite informative and easy to follow.
A: If you're looking for an elementary text, i.e. one that doesn't have a calculus prerequisite, there's Don Berry's Statistics: A Bayesian Perspective.
A: I found an excellent introduction in Gelman and Hill (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models. (Other comments mention it, but it deserves to get upvoted on its own.)
A: Bayesian Modeling Using WinBUGS by Ioannis Ntzoufras, 2009. 
Not only does this book introduce Bayesian methods, but includes everything one needs to immediately start running and diagnosing Bayesian models from simple normal models to generalized linear models to Bayesian hierarchical models. Not impressed yet? An entire chapter on how to set up and run WinBUGS (also useful for OpenBUGS!). Pseudo code and R code side-by-side throughout the book. Everything is illustrated with hands on examples. Also included are a chapter on the predictive distribution and model checking, and a chapter on checking convergence.
It is such an excellent combination of theory, explanation, application, software instruction and code.
A: A really introductory book is Will Kurt's Bayesian Statistics the Fun Way. This approaches the logic of Bayesian statistics at a very high level - going through the basics of probability, and how prior information can combine with data to affect posterior probabilities. As an ecologist who didn't have a robust statistical training in gradschool, I still get bogged down by equations. I found Gelman's book way too difficult for my beginner level, but Will's book is finally helping me wrap my mind around the basic concepts. It has fun, practical examples that would have been a nice per-requisite to read before my grad program.
I plan to work my way through John Kruschke's "Doing Bayesian Data Analysis" as a next stepping stone.

A: Bishop (2006). Pattern Recognition and Machine Learning (PRML)
If you have ever heard about The Elements of Statistical Learning (ESL), PRML can be treated as a bayesian version of it. The book talks about the most important models in statistical/machine learning from linear regression, NN, SVM to graphical models, all in bayesian perspective. The introduction to EM algorithm, variational inference and MCMC are well written.
A: Savage's The Foundations of Statistics helped me understand the why and deeper concepts of Bayesian methods. It's not a how to book, but explains the underlying philosophy of stats.
A: I am stunned at how many introductory books there are.  My recommendation is William Bolstad and James Curran's Introduction to Bayesian Statistics. It has several nice elements to it.
First, it covers a large percentage of the conjugate problems and covers all the standard problems in a sophomore-level test for an introduction to statistics course.  It also covers the case when the prior is not conjugate.
Second, it covers the appropriate Frequentist procedure at each point, explaining both.
Third, it derives the solutions and so allows an understanding of the issues at a deeper level.  It, optionally, allows for rigor.
It covers everything a sophomore-level book would cover, except with underlying math included.  It is an upper-division, lower master's level book due to the included math.

