# Who Are The Bayesians?

As one becomes interested in statistics, the dichotomy "Frequentist" vs. "Bayesian" soon becomes commonplace (and who hasn't read Nate Silver's The Signal and the Noise, anyway?). In talks and introductory courses, the point of view is overwhelmingly frequentist (MLE, $p$ values), but there tends to be a tiny fraction of time dedicated to admire Bayes formula and touch upon the idea of a prior distribution, usually tangentially.

The tone employed to discuss Bayesian statistics oscillates between respect for its conceptual underpinnings, and a hint of skepticism regarding the chasm between lofty objectives, and arbitrariness in the selection of the prior distribution, or eventual use of frequentist maths after all.

Sentences such as "if you are a hard-core Bayesian..." abound.

The question is, Who are the Bayesians today? Are they some select academic institutions where you know that if you go there you will become a Bayesian? If so, are they specially sought after? Are we referring to just a few respected statisticians and mathematicians, and if so who are they?

Do they even exist as such, these pure "Bayesians"? Would they happily accept the label? Is it always a flattering distinction? Are they mathematicians with peculiar slides in meetings, deprived of any $p$ values and confidence intervals, easily spotted on the brochure?

How much of a niche is being a "Bayesian"? Are we referring to a minority of statisticians?

Or is current Bayesian-ism equated with machine learning applications?

... Or even more likely, is Bayesian statistics not so much a branch of statistics, but rather an epistemological movement that transcends the ambit of probability calculations into a philosophy of science? In this regard, all scientists would be Bayesian at heart... but there would be no such thing as a pure Bayesian statistician impermeable to frequentist techniques (or contradictions).

• I would also like to know! My feeling is that "Bayesians" is often a term used by those that doesn't like that type of statistics. I'm a huge fan of Bayesian data analysis, but I don't consider myself a Bayesian, in the same way I don't consider myself a Matrix-algebraist. Aug 13 '15 at 18:19
• The apparent division is, in some ways, imaginary. Sometimes people just like to fall into a them-and-us approach. I get the impression that after a few years nobody cares any more. The "philosophies" are not contradictory of each other. Frequentists do not have a magic recipe for finding good estimators. But given two estimators, they might have a criterion for deciding which estimator is best. (Even then, two frequentists might disagree with each other and use different criteria. But I digress). ... Aug 13 '15 at 20:49
• ... (continued) A hardcore frequentist, in search of a class of estimators from which to select the "best" one, might reasonably decide to consider the class of all Bayesian estimators (i.e. priors) and therefore use the estimator (prior) that is best according their "objective" criterion. Is such a person frequentist (because of how they select the best estimator), or Bayesian (because they consider only Bayesian estimators as candidates)? Does anyone care? I guess many such people call themselves Bayesian, even though they may be wrong in their self-assignment. Aug 13 '15 at 20:49
• Just to note-- MLEs are based too on likelihoodist methods and not purely frequentist. Aug 13 '15 at 22:24
• @Count Some of the literature with which I am familiar (in risk communication and related psychology--Kahneman, Slovic, Tersky, et al.) shows that people do not use mathematically correct procedures to reason about probabilities. For a popular account of some of this, see Kahneman's Thinking, Fast and Slow. The logical implication of your comment, then, is that humans are not "complex lifeforms."
– whuber
Aug 14 '15 at 13:45

I'm going to take your questions in order:

The question is, Who are the Bayesians today?

Anybody who does Bayesian data analysis and self-identifies as "Bayesian". Just like a programmer is someone who programs and self-identifies as a "programmer". A slight difference is that for historical reasons Bayesian has ideological connotations, because of the often heated argument between proponents of "frequentist" interpretations of probability and proponents of "Bayesian" interpretations of probability.

Are they some select academic institutions, where you know that if you go there you will become a Bayesian?

No, just like other parts of statistics you just need a good book (and perhaps a good teacher).

If so, are they specially sought after?

Bayesian data analysis is a very useful tool when doing statistical modeling, which I imagine is a pretty sought-after skill, (even if companies perhaps aren't specifically looking for "Bayesians").

Are we referring to just a few respected statisticians and mathematicians, and if so who are they?

There are many respected statisticians that I believe would call themselves Bayesians, but those are not the Bayesians.

Do they even exist as such, these pure "Bayesians"?

That's a bit like asking "Do these pure programmers exist"? There is an amusing article called 46656 Varieties of Bayesians, and sure there is a healthy argument among "Bayesians" regarding many foundational issues. Just like programmers can argue over the merits of different programming techniques. (BTW, pure programmers program in Haskell).

Would they happily accept the label?

Some do, some don't. When I discovered Bayesian data analysis I thought it was the best since sliced bread (I still do) and I was happy to call myself a "Bayesian" (not least to irritate the p-value people at my department). Nowadays I don't like the term, I think it might alienate people as it makes Bayesian data analysis sound like some kind of cult, which it isn't, rather than a useful method to have in your statistical toolbox.

Is it always a flattering distinction?

Nope! As far as I know, the term "Bayesian" was introduced by the famous statistician Fisher as a derogatory term. Before that it was called "inverse probability" or just "probability".

Are they mathematicians with peculiar slides in meetings, deprived of any p values and confidence intervals, easily spotted on the brochure?

Well, there are conferences in Bayesian statistics, and I don't think they include that many p-values. Whether you'll find the slides peculiar will depend on your background...

How much of a niche is being a "Bayesian"? Are we referring to a minority of statisticians?

I still think a minority of statisticians deal with Bayesian statistics, but I also think the proportion is growing.

Or is current Bayesian-ism equated with machine learning applications?

Nope, but Bayesian models are used a lot in machine learning. Here is a great machine learning book that presents machine learning from a Bayesian/probibalistic perspective: http://www.cs.ubc.ca/~murphyk/MLbook/

Hope that answered most of the questions :)

Update:

[C]ould you please consider adding a list of specific techniques or premises that distinguish Bayesian statistics?

What distinguish Bayesian statistics is the use of Bayesian models :) Here is my spin on what a Bayesian model is:

A Bayesian model is a statistical model where you use probability to represent all uncertainty within the model, both the uncertainty regarding the output but also the uncertainty regarding the input (aka parameters) to the model. The whole prior/posterior/Bayes theorem thing follows on this, but in my opinion, using probability for everything is what makes it Bayesian (and indeed a better word would perhaps just be something like probabilistic model).

Now, Bayesian models can be tricky to fit, and there is a host of different computational techniques that are used for this. But these techniques are not Bayesian in themselves. To namedrop some computational techniques:

• Markov chain Monte Carlo
• Metropolis-Hastings
• Gibbs sampling
• Hamiltonian Monte Carlo
• Variational Bayes
• Approximate Bayesian computation
• Particle filters
• Laplace approximation
• And so on...

Who was the famous statistician who introduced the term 'Bayesian' as derogatory?

It was supposedly Ronald Fisher. The paper When did Bayesian inference become "Bayesian"? gives the history of the term "Bayesian".

• Oh wow, I remember you from the post about celebrity heights on Andrew Gelman's blog! I'm looking forward to reading, "46656 Varieties of Bayesians". Thank you for a good answer! Aug 14 '15 at 0:52
• Very good! I like that you dropped the word 'cult'. I was hesitant lest anybody was offended. Some of my questions were just meant to be prompts... in the end, I am trying to learn about statistics, and I was curious to understand the dichotomy from the inside. Aug 14 '15 at 1:17
• A comment: there are many things that are labeled as "Bayesian" and people tend to confuse them (and do in this very Q&A!). A non complete list: The Bayesian Brain hypothesis (a brain is basically doing Bayesian stats), Bayesian philosophy of science, Bayesian statistics, Bayesian view of probability, Computational methods for doing Bayesian statistics, etc. Surely many of these are related (say Bayes. probability and Bayes. stats), but you don't have to buy them all! E.g. I think the Bayesian Brain is highly suspect but embrace Bayesian statistics as a useful and practical technique. Aug 14 '15 at 20:57
• Great post! One thing, however, that I would disagree with is your answer to the "Are they some select academic institutions, where you know that if you go there you will become a Bayesian?" question. If you go to Duke's stat department, you will become a Bayesian. Aug 17 '15 at 13:51
• Man, if I got an upvote for every question I've answered here I would have... 12 upvotes :) Aug 18 '15 at 14:00

Bayesians are people who define probabilities as a numerical representation of the plausibility of some proposition. Frequentists are people who define probabilities as representing long run frequencies. If you are only happy with one or other of these definitions then you are either a Bayesian or a frequentist. If you are happy with either, and use the most appropriate definition for the task at hand, then you are a statistician! ;o) Basically, it boils down to the definition of a probability, and I would hope that most working statisticians would be able to see the benefits and disadvantages of both approaches.

hint of skepticism regarding the chasm between lofty objectives, and arbitrariness in the selection of the prior distribution, or eventual use of frequentist maths after all.

The skepticism also goes in the other direction. Frequentism was invented with the lofty objective of eliminating the subjectivity of existing thought on probability and statistics. However, the subjectivity is still there (for example in determining the appropriate level of significance in hypothesis testing), but it is just not made explicit, or often just ignored.

• This I don't understand. You can define probability as representing long run frequencies, yet believe in a hypothesis only when its P(H|O) is high and know that P(O|H) (p-value) tells you little. (If you live long enough with enough introspection, you can directly count the frequency of having been right.) Aug 15 '15 at 10:30
• Frequentists cannot assign a value to P(H|O) as the truth of a particular hypothesis has no long run frequency, it is either true or it isn't. As a result, we can only attach probabilities to some (possibly fictitious) population of experiments from which the one we actually observed was drawn, or "reject H0" or "fail to reject H0" at a particular significance level. Unfortunately either approach leaves possibilities for misinterpretation as what we actually want from the test is exactly P(H|O). Both approaches have their uses, but it is important to understand their limitations. Aug 17 '15 at 7:27
• Is there any other field of math whose practitioners hold themselves hostage to philosophy? Regardless, in practice, essentially the same questions come up again and again. E.g., "did this person commit murder." The unique identity of the accused is irrelevant (just as we ignore the physical details of a particular die roll). Given the thousands of murders committed each year (and the thousands more innocently accused), any set of circumstances will likely have occurred more than once. What isn't frequentist about deciding someone's guilt? Yet to use p-value would be a grave injustice. Aug 18 '15 at 14:08

Andrew Gelman, for example, a professor of statistics and political science at Columbia University, is a prominent Bayesian.

I suspect the most of ISBA fellows would probably consider themselves Bayesians as well.

In general, the following research topics typically reflect a Bayesian approach. If you read papers about them, it is likely the authors would describe themselves as "Bayesian"

• Markov-Chain Monte Carlo
• Variational Bayesian Methods (the name gives that one away)
• Particle Filtering
• Probabilistic programming
• Small note: particle filtering is not just for Bayesians! I worked under a professor at Berkeley, in which we used particle filters for the E step of an MCEM algorithm. But yes, particle filters are typically used by Bayesians. Aug 13 '15 at 18:42
• If you're going to pay the computational price, why not get the philosophical consistency? Aug 13 '15 at 18:52
• Gelman isn't "hard core" though. As far as I can tell, he sees Bayesian statistics as something that has proven its worth practically, and he is definitely interested in frequentist properties of Bayesian procedures. Aug 13 '15 at 18:56
• It should be noted that Markov-Chain Monte Carlo is not directly related to Bayesian statistics, in the same was as numerical optimization is not directly related to Maximum likelihood estimation... Aug 14 '15 at 11:06
• I think it is also worth noting that Andrew Gelman has written that he doesn't think it is at all meaningful to label a person as a "Bayesian" or "frequentist". But rather, certain TECHNIQUES may be so labeled. He thinks it is counterproductive to arbitrarily divide statisticians into one camp or the other, because both methodologies have different strengths and weaknesses in different contexts. Aug 21 '15 at 18:49

Today, we're all Bayesians, but there's a world beyond these two camps: algorithmic probability. I'm not sure what's the standard reference on this subject, but there's this beautiful paper by Kolmogorov on algorithmic complexity: A. N. Kolmogorov, Three approaches to the definition of the concept “quantity of information”, Probl. Peredachi Inf., 1965, Volume 1, Issue 1, 3–11. I'm sure there's an English translation.

In this paper he defines the quantity of information in three ways: combinatorial, probabilistic and (new) algorithmic. Combinatorial directly maps to frequentist, Probabilist doesn't directly correspond to Bayesian, but it's compatible with it.

UPDATE: If you're interested in the philosophy of the probability then I want to point to a very interesting work "The origins and legacy of Kolmogorov’s Grundbegriffe" by Glenn Shafer and Vladimir Vovk. We sort of forgot everything before Kolmogorov, and there was a lot going on before his seminal work. On the other hand, we don't know much about his philosophical views. It's generally thought that he was a frequentist, for instance. The reality's that he lived in Soviet Union in 1930', where it was quite dangerous to venture into philosophy, literally, you could get in existential trouble, which some scientist did (ended up in GULAG prisons). So, he was sort of forced to implicitly indicate that he was a frequentist. I think that in reality he was not just a mathematician, but he was a scientist, and had a complex view of applicability of probability theory to reality.

There's also another paper by Vovk on Kolmogorov's algorithmic approach to randomness: Kolmogorov’s contributions to the foundations of probability

Vovk has created a game-theoretic approach to probabilities - also very interesting.

UPDATE 2: Here's a Bayesian, actually, a professor from one of the universities in Washington, DC. He was trying to make a point that we should elect politicians who update their beliefs based on experiences, new observations. Here $P(B|E)$ is the posterior belief $B$, after the new experience $E$; $P(E|B)$ is the prior. He was trying to explain this to Colbert/Stuart "Rally for Fear" participants.

UPDATE 3:

I also wanted to point to something in Kolmogorov's original work that's not commonly known for some reason (or easily forgotten) by practitioners. He had a section about connecting the theory to reality. In particular, he set two conditions for using the theory:

• A. if you repeat the experiment many times then the frequency of occurrence will differ by only a small amount from the probability, practically certainly
• B. If probability is very small, then if you conduct the experiment only once then you can be practically certain that the event will not occur

There are different interpretations of these conditions, but most people would agree that these are not the pure frequentist's views. Kolmogorov declared that he follows von Mises' approach to certain extent, but he seemed to indicate that things are not as simple as it may appear. I often think of condition B, and can't come to a stable conclusion, it looks slightly different every time I think about it.

• Is your first hyperlink what you intended? Aug 13 '15 at 23:03
• @AntoniParellada, it's intended to be a joke :) Aug 14 '15 at 1:37
• Completely over my head... And it's probably funny, given the inane nature of the hyperlinked discussion... So sorry I missed it... Aug 14 '15 at 1:47
• "McCain to Georgian President: "Today, We Are All Georgians" Haha, this is funny. Aug 14 '15 at 10:26
• @Aksakal Kolmogorov in English Aug 16 '15 at 12:49

The most "hard core" Bayesian that I know of is Edwin Jaynes, deceased in 1998. I'd expect further "hard core" Bayesians to be found among his pupils, especially the posthumous co-author of his main work Probability Theory: The Logic of Science, Larry Bretthorst. Other notable historic Bayesians include Harold Jeffreys and and Leonard Savage. While I don't have a complete overview of the field, my impression is that the more recent popularity of Bayesian methods (especially in machine learning) is not due to deep philosophical conviction, but the pragmatic position that Bayesian methods have proved useful in many applications. I think typical for this position is Andrew Gelman.

• It sounds a bit like a romantic idea. Norman Rockwell of statistics? Aug 13 '15 at 21:31
• @AntoniParellada, I have no idea what you mean by that... Aug 14 '15 at 3:12
• Jaynes and Jeffreys were who I had in mind too. A great essay is "Where Do We Stand On Maximum Entropy?" Aug 14 '15 at 7:26
• Hmmm, I always read Jaynes as being very pragmatic about Bayes. Aug 14 '15 at 12:55

I don't know who the Bayesians are (although I suppose I should have a prior distribution for that), but I do know who they are not.

To quote the eminent, now departed Bayesian, D.V. Lindley, "there is no one less Bayesian than an empirical Bayesian". Empirical Bayes section of Bayesian Methods: A Social and Behavioral Sciences Approach, Second Edition by Jeff Gill. Meaning I suppose that even "Frequentists" think about what model makes sense (choice of a model form in some sense constitutes a prior), as opposed to empirical Bayesians who are totally mechanical about everything.

I think that in practice there is not that much difference in the results of statistical analysis performed by top echelon Bayesians and Frequentists. What is scary is when you see a low quality statistician who tries to rigidly pattern himself (never observed it with a female) after his ideological role model with absolute ideological purity, and approach analysis exactly as he thinks his role model would, but without the quality of thought and judgment the role model has. That can result in very bad analysis and recommendations. I think ultra-hard core, but low quality, ideologs are much more common among Bayesians than Frequentists. This particularly applies in Decision Analysis.

• It's nice to point to some rigidities with humor. Ty Aug 13 '15 at 21:34

I'm probably too late to this discussion for anyone to notice this, but I think it is a shame that no-one has pointed out the fact that the most important difference between Bayesian and Frequentist approaches is that the Bayesians (mostly) use methods that respect the likelihood principle whereas Frequentists almost invariably do not. The likelihood principle says that the evidence relevant to the statistical model parameter of interest in entirely contained in the relevant likelihood function.

Frequentists who care about statistical theory or philosophy should be far more concerned by arguments about the validity of the likelihood principle than about arguments over the distinction between frequency and partial belief interpretations of probability and about the desirability of prior probabilities. While it is possible for different interpretations of probability to coexist without conflict and for some people to choose to supply a prior without requiring others to do so, if the likelihood principle is true in either a positive or normative sense then many Frequentist methods lose their claims to optimality. Frequentist attacks on the likelihood principle are vehement because that principle undermines their statistical world-view, but mostly those attacks miss their mark (http://arxiv.org/abs/1507.08394).

Bayesians derive the probability distribution of outcomes of interest given prior belief / prior information. To a Bayesian this distribution (and its summaries) are what most people will be interested in. Contrast with typical "frequentist" results that tell you what the chance of seeing results as or more extreme than those observed given that the null hypothesis is true (p-value) or interval estimates for the parameter of interest, 95% of which would contain the true value if you could do repeated sampling (confidence interval).

Bayesian prior distributions are contentious because they are YOUR prior. There is no "correct" prior. Most pragmatic Bayesians look for external evidence that can be used for priors and then discount or modify this based on what is expected to be "reasonable" for the particular case. For example, sceptical priors may have a "lump" of probability on a null case - "How good would the data need to be to make me change my mind / change current practice?" Most will also look at robustness of inferences to different priors.

There are a group of Bayesians who look into "reference" priors that allow them to construct inferences that are not "influenced" by prior belief and so they get probabilistic statements and interval estimates that have "frequentist" properties.

There are also a group of "Hardcore Bayesians" who might advocate not choosing a model (all models are wrong), and who might argue that exploratory analysis is bound to influence your priors and so shouldn't be done. There are few that radical though...

In most fields of statistics you'll find Bayesian analyses and practitioners. Just as you'll find some folks who prefer non-parametrics...

• I think I understand Bayesian statistics better after reading your post. I wonder if you can tie it up to the actual question to wrap it up into an outstanding answer... It was along the lines of Bayesians being a specific group of people with names, or mathematical departments known for their bias towards Bayes approach to statistics, etc. Aug 14 '15 at 12:45
• There are many individuals and academic departments that have favoured Bayesian statistics now and in the past. It's hard to single out any one in particular. If you're interested more then I'd recommend looking at ISBA bayesian.org. Aug 14 '15 at 12:58
• A few names to look out for: Don Berry, Jim Berger, David Draper, Merlise Clyde, Mike West, David Spiegelhalter, Peter Thall... Aug 14 '15 at 12:58
• Yes, someone else posted the link, and I actually went through the alphabetic list looking for patterns... I couldn't find any, which is not surprising since I'm not a statistician. I guess the idea boils down to, Is Bayes a lofty, pure idea people like to claim adherence to, or is it (still) a well-defined, day-to-day way of practicing applied statistics in contradistinction to frequentism - the latter one not too "sexy" sounding to attach your name to, but possibly more practical? Aug 14 '15 at 13:03
• In answer to your last question, it is both. It is definitely a philosophical approach. It complements the scientific method that says that we observe what is (prior information), hypothesise, experiment, synthesise and update our current knowledge, which becomes tomorrow's prior. But it's also a statistical method of analysis that can be applied to an individual case. Aug 14 '15 at 13:13

Just to take up your last question (so I'm not after a prize!), about a link between a Bayesian/Frequentist approach and one's epistemological position, the most interesting author I've come upon is Deborah Mayo. A good starting point is this 2010 exchange between Mayo and Andrew Gelman (who emerges here as a somewhat heretical Bayesian). Mayo later published a detailed response to the Gelman & Shalizi paper here.

A subset of all Bayesians, i.e. those Bayesians who bothered to send an email, is listed here.

• I found there two stat professors who identify themselves as Bayesians. This must be a good list then. Aug 13 '15 at 21:03
• @Aksakal I think that's more a reflection of the fact that statisticians come from a variety of backgrounds. If the criterion is "people who have published in major stats journals" then many dozens of those names count, whatever the name of the department they're in. I recognized quite a large number just scanning down the list. Aug 14 '15 at 1:22
• @Aksakal I don't understand your point. There are 2 (perhaps 3) statistics professors in the first 5 people on that list. Aug 15 '15 at 1:24
• @jaradniemi, I recalled two my professors who are openly Bayesians, then found them in the list. This makes me think that the list is probably representative. Aug 15 '15 at 1:27

I would call Bruno de Finetti and L. J. Savage Bayesians. They worked on its philosophical foundations.

• Since De Finetti (notice the spelling) died 30 years ago and Savage 44 years ago, they scarcely could be recognized as answering "Who are the Bayesians today?", unless they somehow have risen as zombies and are publishing pseudonymously.
– whuber
Aug 14 '15 at 3:56
• @whuber... It sounds (from the outside) more like a nice, lofty idea... akin to thinking of oneself as evidence-based, constantly updating our view of the world based on our priors and the evidence collected. Bayes as epistemology... rather than a strict adherence to a "different" set of statistical techniques.... Aug 14 '15 at 12:07

For understanding the foundational debate between frequentists and Bayesians, it would be hard to find a more authoritative voice than Bradley Efron.

This topic has been a theme he has touched on numerous times in his career, but personally I found one of his older papers helpful: Controversies in the Foundations of Statistics (this one won an award for expository excellence).