3
$\begingroup$

In a textbook Probability Theory: The Logic of Science written by E. T. Jaynes and others, on page 13 it reads that:

For many years, there has been controversy over ‘frequentist’ versus ‘Bayesian’ methods of inference, in which the writer has been an outspoken partisan on the Bayesian side.

From this answer, if I am not wrong, I conjecture that any task which suits a frequentist approach can be solved also in a Bayesian way, and vice versa.

I later learned that the rise of MCMC algorithms in the 80's lead to an exponential rise in the use of Bayes in methods as complex models built through hierarchical distributions suddenly were tractable(reference: Mixtures of Conjugate Priors and MCMC).

Then my question is that if the Bayesian approach is better than the frequentist (1. Jaynes taught me that in that book; 2. for instance more parameters in models like the hidden Markov model can learn more information and then be better to reach the reality, the posterior distribution than the Markov model; 3) more reasons from this answer to this question: What is the importance of probabilistic machine learning?) and it has been tractable, then why frequentist methods still dominate the machine learning field?

In the answers to this question I learned why frequentist is better practically. But I wonder if Bayesian can be as practical as frequentist approaches? How can it be?

$\endgroup$
3
  • 1
    $\begingroup$ Both Bayesian and frequentist statistics are huge areas on their own, so asking for "the bottlenecks" will require a book with multiple volumes to answer. Can you narrow it down? $\endgroup$ – Maarten Buis Oct 5 '18 at 14:57
  • $\begingroup$ @MaartenBuis Sorry for that. Only some points are enough and details are not needed. $\endgroup$ – Lerner Zhang Oct 5 '18 at 22:32
  • 1
    $\begingroup$ Might want to compare this question / answer: stats.stackexchange.com/questions/20558/… $\endgroup$ – jbowman Oct 5 '18 at 22:38
10
$\begingroup$

MCMC is just an algorithm to sample from a distribution. On its own it is neither frequentist nor Bayesian. However, this algorithm has proven to be extremely useful for Bayesians. It has made it feasible to sample from the posterior distribution, and with those samples in hand, describe the posterior distribution. Before its use, real use of Bayesian statistics was mostly limited to models with conjugate priors. Meaning that before MCMC Bayesian statistics was more a theoretical, but unpractical, area of statistics. So most statistician would associate MCMC with Bayesian rather than frequentist statistics.

Is see both approaches as necessarily flawed approximations to answer to an unanswerable question: How do I give an answer when I don't have all the information? In the way I use statistics, the difference between the outcomes of these two methods is negligible. So I use pragmatic reasons to choose my method: unless Bayesian statistics has a clear added value, I will use frequentist because it is easier to "sell" my articles to journals. The potential added value of a Bayesian analysis would for example be if I have a prior that I really want to include in my analysis, and that prior is strong enough to influence my results. Especially, the latter is a fairly hard condition to meet in the type of analyses I do and the type of data I usually work with. So in the end I almost never use Bayesian statistics.

Some claim that the added value of the Bayesian is that the inference is more intuitive, and I kinda get their argument. However, it ignores the fact that most of my audience is not trained in Bayesian statistics. So until the majority is trained in Bayesian statistics, its potential cannot be realized. This is a catch 22: people are trained in frequentist statistics because that is what everybody uses, and everybody uses frequentists statistics because that is what everybody is trained in. However, we have to work with the world we live in, and not with the world we would want to live in.

$\endgroup$
6
  • 2
    $\begingroup$ Thanks for your awesome answer! But it seems that I have not fully understood the difference between these to approaches. I will update the question later. $\endgroup$ – Lerner Zhang Oct 5 '18 at 12:35
  • 1
    $\begingroup$ More than two years later I updated the question. Could you please help check if it makes sense now? Thanks. $\endgroup$ – Lerner Zhang Jan 16 at 8:22
  • 1
    $\begingroup$ I find more and more scholars become secular Bayesian, for instance this guy. $\endgroup$ – Lerner Zhang Jan 16 at 8:39
  • 2
    $\begingroup$ Here you can find people who are not secular Bayesians! $\endgroup$ – kjetil b halvorsen Jan 16 at 12:47
  • 2
    $\begingroup$ The fact that most of the audience is not trained in Bayesian statistics doesn't hold water as an argument against Bayes. Much of the audience is trained in frequentist statistics yet still don't understand frequentist inference at all. $\endgroup$ – Frank Harrell Jan 16 at 13:11
0
$\begingroup$

A position answer.

I learned from this blog: https://vasishth.github.io/bayescogsci/ by Prof. Dr. Shravan Vasishth that

In recent years, Bayesian methods have come to be widely adopted in all areas of science. This is in large part due to the development of sophisticated software for probabilisic programming; a recent example is the astonishing computing capability afforded by the language, Stan. However, the underlying theory needed to use such computational tools sensibly is often inaccessible because end-users don't necessarily have the statistical and mathematical background to read the primary textbooks (such as Gelman et al's classic Bayesian Data Analysis, 3rd edition).

I thought the only thing that hinders it from being practical is the deep learning curve which is much deeper than deep learning. For instance, in industry, TF probability sets a higher bar.

And I think this statement would be right: Everything that Works Works Because it's Bayesian.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.