In a textbook Probability Theory: The Logic of Science written by E. T. Jaynes and others, on page 13 it reads that:

For many years, there has been controversy over ‘frequentist’ versus ‘Bayesian’ methods of inference, in which the writer has been an outspoken partisan on the Bayesian side.

While comparing the structures of the Markov Model and Hidden Markov Model, and discriminating the Markov Chain Monte Carlo (MCMC) approximation from the variational approximation, an idea swims into my mind that the introduction of a hidden layer into Markov Chain is the transition from the frequentist(Markov chain) to Bayesian(Hidden Markov Chain) and MCMC is an frequentist approach but the variational approximation belongs to the Bayesian.

From this answer, if I am not wrong, I conjecture that any task which suits a frequentist approach can be solved also in a Bayesian way, and the vice versa.

Then my question is that if the Bayesian approach is better than the frequentist(1. Jaynes taught me that in that book; 2. for instance more parameters in models like hidden markov model can learn more information and then be better to reach the reality, the posterior distribution than the markov model), then why we still need frequentist methods like MCMC?

In the answers of this question I learned why frequentist is better practically. But I wonder if Bayesian can be as practical as frequentist approaches? How can it be?

  • $\begingroup$ Both Bayesian and frequentist statistics are huge areas on their own, so asking for "the bottlenecks" will require a book with multiple volumes to answer. Can you narrow it down? $\endgroup$ – Maarten Buis Oct 5 '18 at 14:57
  • $\begingroup$ @MaartenBuis Sorry for that. Only some points are enough and details are not needed. $\endgroup$ – Lerner Zhang Oct 5 '18 at 22:32
  • 1
    $\begingroup$ Might want to compare this question / answer: stats.stackexchange.com/questions/20558/… $\endgroup$ – jbowman Oct 5 '18 at 22:38

MCMC is just an algorithm to sample from a distribution. On its own it is neither frequentist nor Bayesian. However, this algorithm has proven to be extremely useful for Bayesians. It has made it feasible to sample from the posterior distribution, and with those samples in hand, describe the posterior distribution. Before its use, real use of Bayesian statistics was mostly limited to models with conjugate priors. Meaning that before MCMC Bayesian statistics was more a theoretical, but unpractical, area of statistics. So most statistician would associate MCMC with Bayesian rather than frequentist statistics.

Is see both approaches as necessarily flawed approximations to answer to an unanswerable question: How do I give an answer when I don't have all the information? In the way I use statistics, the difference between the outcomes of these two methods is negligible. So I use pragmatic reasons to choose my method: unless Bayesian statistics has a clear added value, I will use frequentist because it is easier to "sell" my articles to journals. The potential added value of a Bayesian analysis would for example be if I have a prior that I really want to include in my analysis, and that prior is strong enough to influence my results. Especially, the latter is a fairly hard condition to meet in the type of analyses I do and the type of data I usually work with. So in the end I almost never use Bayesian statistics.

Some claim that the added value of the Bayesian is that the inference is more intuitive, and I kinda get their argument. However, it ignores the fact that most of my audience is not trained in Bayesian statistics. So until the majority is trained in Bayesian statistics, its potential cannot be realized. This is a catch 22: people are trained in frequentist statistics because that is what everybody uses, and everybody uses frequentists statistics because that is what everybody is trained in. However, we have to work with the world we live in, and not with the world we would want to live in.

  • 2
    $\begingroup$ Thanks for your awesome answer! But it seems that I have not fully understood the difference between these to approaches. I will update the question later. $\endgroup$ – Lerner Zhang Oct 5 '18 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.