Who Are The Bayesians?

Question

As one becomes interested in statistics, the dichotomy "Frequentist" vs. "Bayesian" soon becomes commonplace (and who hasn't read Nate Silver's The Signal and the Noise, anyway?). In talks and introductory courses, the point of view is overwhelmingly frequentist (MLE, $p$ values), but there tends to be a tiny fraction of time dedicated to admire Bayes formula and touch upon the idea of a prior distribution, usually tangentially.

The tone employed to discuss Bayesian statistics oscillates between respect for its conceptual underpinnings, and a hint of skepticism regarding the chasm between lofty objectives, and arbitrariness in the selection of the prior distribution, or eventual use of frequentist maths after all.

Sentences such as "if you are a hard-core Bayesian..." abound.

The question is, Who are the Bayesians today? Are they some select academic institutions where you know that if you go there you will become a Bayesian? If so, are they specially sought after? Are we referring to just a few respected statisticians and mathematicians, and if so who are they?

Do they even exist as such, these pure "Bayesians"? Would they happily accept the label? Is it always a flattering distinction? Are they mathematicians with peculiar slides in meetings, deprived of any $p$ values and confidence intervals, easily spotted on the brochure?

How much of a niche is being a "Bayesian"? Are we referring to a minority of statisticians?

Or is current Bayesian-ism equated with machine learning applications?

... Or even more likely, is Bayesian statistics not so much a branch of statistics, but rather an epistemological movement that transcends the ambit of probability calculations into a philosophy of science? In this regard, all scientists would be Bayesian at heart... but there would be no such thing as a pure Bayesian statistician impermeable to frequentist techniques (or contradictions).

Answer 1

I'm going to take your questions in order:

The question is, Who are the Bayesians today?

Anybody who does Bayesian data analysis and self-identifies as "Bayesian". Just like a programmer is someone who programs and self-identifies as a "programmer". A slight difference is that for historical reasons Bayesian has ideological connotations, because of the often heated argument between proponents of "frequentist" interpretations of probability and proponents of "Bayesian" interpretations of probability.

Are they some select academic institutions, where you know that if you go there you will become a Bayesian?

No, just like other parts of statistics you just need a good book (and perhaps a good teacher).

If so, are they specially sought after?

Bayesian data analysis is a very useful tool when doing statistical modeling, which I imagine is a pretty sought-after skill, (even if companies perhaps aren't specifically looking for "Bayesians").

Are we referring to just a few respected statisticians and mathematicians, and if so who are they?

There are many respected statisticians that I believe would call themselves Bayesians, but those are not the Bayesians.

Do they even exist as such, these pure "Bayesians"?

That's a bit like asking "Do these pure programmers exist"? There is an amusing article called 46656 Varieties of Bayesians, and sure there is a healthy argument among "Bayesians" regarding many foundational issues. Just like programmers can argue over the merits of different programming techniques. (BTW, pure programmers program in Haskell).

Would they happily accept the label?

Some do, some don't. When I discovered Bayesian data analysis I thought it was the best since sliced bread (I still do) and I was happy to call myself a "Bayesian" (not least to irritate the p-value people at my department). Nowadays I don't like the term, I think it might alienate people as it makes Bayesian data analysis sound like some kind of cult, which it isn't, rather than a useful method to have in your statistical toolbox.

Is it always a flattering distinction?

Nope! As far as I know, the term "Bayesian" was introduced by the famous statistician Fisher as a derogatory term. Before that it was called "inverse probability" or just "probability".

Are they mathematicians with peculiar slides in meetings, deprived of any p values and confidence intervals, easily spotted on the brochure?

Well, there are conferences in Bayesian statistics, and I don't think they include that many p-values. Whether you'll find the slides peculiar will depend on your background...

How much of a niche is being a "Bayesian"? Are we referring to a minority of statisticians?

I still think a minority of statisticians deal with Bayesian statistics, but I also think the proportion is growing.

Or is current Bayesian-ism equated with machine learning applications?

Nope, but Bayesian models are used a lot in machine learning. Here is a great machine learning book that presents machine learning from a Bayesian/probibalistic perspective: http://www.cs.ubc.ca/~murphyk/MLbook/

Hope that answered most of the questions :)

Update:

[C]ould you please consider adding a list of specific techniques or premises that distinguish Bayesian statistics?

What distinguish Bayesian statistics is the use of Bayesian models :) Here is my spin on what a Bayesian model is:

A Bayesian model is a statistical model where you use probability to represent all uncertainty within the model, both the uncertainty regarding the output but also the uncertainty regarding the input (aka parameters) to the model. The whole prior/posterior/Bayes theorem thing follows on this, but in my opinion, using probability for everything is what makes it Bayesian (and indeed a better word would perhaps just be something like probabilistic model).

Now, Bayesian models can be tricky to fit, and there is a host of different computational techniques that are used for this. But these techniques are not Bayesian in themselves. To namedrop some computational techniques:

• Markov chain Monte Carlo
  • Metropolis-Hastings
  • Gibbs sampling
  • Hamiltonian Monte Carlo
• Variational Bayes
• Approximate Bayesian computation
• Particle filters
• Laplace approximation
• And so on...

Who was the famous statistician who introduced the term 'Bayesian' as derogatory?

It was supposedly Ronald Fisher. The paper When did Bayesian inference become "Bayesian"? gives the history of the term "Bayesian".

Answer 2

Bayesians are people who define probabilities as a numerical representation of the plausibility of some proposition. Frequentists are people who define probabilities as representing long run frequencies. If you are only happy with one or other of these definitions then you are either a Bayesian or a frequentist. If you are happy with either, and use the most appropriate definition for the task at hand, then you are a statistician! ;o) Basically, it boils down to the definition of a probability, and I would hope that most working statisticians would be able to see the benefits and disadvantages of both approaches.

hint of skepticism regarding the chasm between lofty objectives, and arbitrariness in the selection of the prior distribution, or eventual use of frequentist maths after all.

The skepticism also goes in the other direction. Frequentism was invented with the lofty objective of eliminating the subjectivity of existing thought on probability and statistics. However, the subjectivity is still there (for example in determining the appropriate level of significance in hypothesis testing), but it is just not made explicit, or often just ignored.

Answer 3

Today, we're all Bayesians, but there's a world beyond these two camps: algorithmic probability. I'm not sure what's the standard reference on this subject, but there's this beautiful paper by Kolmogorov on algorithmic complexity: A. N. Kolmogorov, Three approaches to the definition of the concept “quantity of information”, Probl. Peredachi Inf., 1965, Volume 1, Issue 1, 3–11. I'm sure there's an English translation.

In this paper he defines the quantity of information in three ways: combinatorial, probabilistic and (new) algorithmic. Combinatorial directly maps to frequentist, Probabilist doesn't directly correspond to Bayesian, but it's compatible with it.

UPDATE: If you're interested in the philosophy of the probability then I want to point to a very interesting work "The origins and legacy of Kolmogorov’s Grundbegriffe" by Glenn Shafer and Vladimir Vovk. We sort of forgot everything before Kolmogorov, and there was a lot going on before his seminal work. On the other hand, we don't know much about his philosophical views. It's generally thought that he was a frequentist, for instance. The reality's that he lived in Soviet Union in 1930', where it was quite dangerous to venture into philosophy, literally, you could get in existential trouble, which some scientist did (ended up in GULAG prisons). So, he was sort of forced to implicitly indicate that he was a frequentist. I think that in reality he was not just a mathematician, but he was a scientist, and had a complex view of applicability of probability theory to reality.

There's also another paper by Vovk on Kolmogorov's algorithmic approach to randomness: Kolmogorov’s contributions to the foundations of probability

Vovk has created a game-theoretic approach to probabilities - also very interesting.

UPDATE 2: Here's a Bayesian, actually, a professor from one of the universities in Washington, DC. He was trying to make a point that we should elect politicians who update their beliefs based on experiences, new observations. Here $P(B|E)$ is the posterior belief $B$, after the new experience $E$; $P(E|B)$ is the prior. He was trying to explain this to Colbert/Stuart "Rally for Fear" participants.

UPDATE 3:

I also wanted to point to something in Kolmogorov's original work that's not commonly known for some reason (or easily forgotten) by practitioners. He had a section about connecting the theory to reality. In particular, he set two conditions for using the theory:

• A. if you repeat the experiment many times then the frequency of occurrence will differ by only a small amount from the probability, practically certainly
• B. If probability is very small, then if you conduct the experiment only once then you can be practically certain that the event will not occur

There are different interpretations of these conditions, but most people would agree that these are not the pure frequentist's views. Kolmogorov declared that he follows von Mises' approach to certain extent, but he seemed to indicate that things are not as simple as it may appear. I often think of condition B, and can't come to a stable conclusion, it looks slightly different every time I think about it.

Answer 4

Andrew Gelman, for example, a professor of statistics and political science at Columbia University, is a prominent Bayesian.

I suspect the most of ISBA fellows would probably consider themselves Bayesians as well.

In general, the following research topics typically reflect a Bayesian approach. If you read papers about them, it is likely the authors would describe themselves as "Bayesian"

• Markov-Chain Monte Carlo
• Variational Bayesian Methods (the name gives that one away)
• Particle Filtering
• Probabilistic programming

Answer 5

The most "hard core" Bayesian that I know of is Edwin Jaynes, deceased in 1998. I'd expect further "hard core" Bayesians to be found among his pupils, especially the posthumous co-author of his main work Probability Theory: The Logic of Science, Larry Bretthorst. Other notable historic Bayesians include Harold Jeffreys and and Leonard Savage. While I don't have a complete overview of the field, my impression is that the more recent popularity of Bayesian methods (especially in machine learning) is not due to deep philosophical conviction, but the pragmatic position that Bayesian methods have proved useful in many applications. I think typical for this position is Andrew Gelman.

Answer 6

I don't know who the Bayesians are (although I suppose I should have a prior distribution for that), but I do know who they are not.

To quote the eminent, now departed Bayesian, D.V. Lindley, "there is no one less Bayesian than an empirical Bayesian". Empirical Bayes section of Bayesian Methods: A Social and Behavioral Sciences Approach, Second Edition by Jeff Gill. Meaning I suppose that even "Frequentists" think about what model makes sense (choice of a model form in some sense constitutes a prior), as opposed to empirical Bayesians who are totally mechanical about everything.

I think that in practice there is not that much difference in the results of statistical analysis performed by top echelon Bayesians and Frequentists. What is scary is when you see a low quality statistician who tries to rigidly pattern himself (never observed it with a female) after his ideological role model with absolute ideological purity, and approach analysis exactly as he thinks his role model would, but without the quality of thought and judgment the role model has. That can result in very bad analysis and recommendations. I think ultra-hard core, but low quality, ideologs are much more common among Bayesians than Frequentists. This particularly applies in Decision Analysis.

Answer 7

I'm probably too late to this discussion for anyone to notice this, but I think it is a shame that no-one has pointed out the fact that the most important difference between Bayesian and Frequentist approaches is that the Bayesians (mostly) use methods that respect the likelihood principle whereas Frequentists almost invariably do not. The likelihood principle says that the evidence relevant to the statistical model parameter of interest in entirely contained in the relevant likelihood function.

Frequentists who care about statistical theory or philosophy should be far more concerned by arguments about the validity of the likelihood principle than about arguments over the distinction between frequency and partial belief interpretations of probability and about the desirability of prior probabilities. While it is possible for different interpretations of probability to coexist without conflict and for some people to choose to supply a prior without requiring others to do so, if the likelihood principle is true in either a positive or normative sense then many Frequentist methods lose their claims to optimality. Frequentist attacks on the likelihood principle are vehement because that principle undermines their statistical world-view, but mostly those attacks miss their mark (http://arxiv.org/abs/1507.08394).

Answer 8

You may believe you're a Bayesian, but you're probably wrong... http://www.rmm-journal.de/downloads/Article_Senn.pdf

Bayesians derive the probability distribution of outcomes of interest given prior belief / prior information. To a Bayesian this distribution (and its summaries) are what most people will be interested in. Contrast with typical "frequentist" results that tell you what the chance of seeing results as or more extreme than those observed given that the null hypothesis is true (p-value) or interval estimates for the parameter of interest, 95% of which would contain the true value if you could do repeated sampling (confidence interval).

Bayesian prior distributions are contentious because they are YOUR prior. There is no "correct" prior. Most pragmatic Bayesians look for external evidence that can be used for priors and then discount or modify this based on what is expected to be "reasonable" for the particular case. For example, sceptical priors may have a "lump" of probability on a null case - "How good would the data need to be to make me change my mind / change current practice?" Most will also look at robustness of inferences to different priors.

There are a group of Bayesians who look into "reference" priors that allow them to construct inferences that are not "influenced" by prior belief and so they get probabilistic statements and interval estimates that have "frequentist" properties.

There are also a group of "Hardcore Bayesians" who might advocate not choosing a model (all models are wrong), and who might argue that exploratory analysis is bound to influence your priors and so shouldn't be done. There are few that radical though...

In most fields of statistics you'll find Bayesian analyses and practitioners. Just as you'll find some folks who prefer non-parametrics...

Answer 9

Just to take up your last question (so I'm not after a prize!), about a link between a Bayesian/Frequentist approach and one's epistemological position, the most interesting author I've come upon is Deborah Mayo. A good starting point is this 2010 exchange between Mayo and Andrew Gelman (who emerges here as a somewhat heretical Bayesian). Mayo later published a detailed response to the Gelman & Shalizi paper here.

Answer 10

I would call Bruno de Finetti and L. J. Savage Bayesians. They worked on its philosophical foundations.

Answer 11

A subset of all Bayesians, i.e. those Bayesians who bothered to send an email, is listed here.

Answer 12

For understanding the foundational debate between frequentists and Bayesians, it would be hard to find a more authoritative voice than Bradley Efron.

This topic has been a theme he has touched on numerous times in his career, but personally I found one of his older papers helpful: Controversies in the Foundations of Statistics (this one won an award for expository excellence).