If Bayesian is better than frequentist(and it's tractable) then how can it be as practical?

Question

In a textbook Probability Theory: The Logic of Science written by E. T. Jaynes and others, on page 13 it reads that:

For many years, there has been controversy over ‘frequentist’ versus ‘Bayesian’ methods of inference, in which the writer has been an outspoken partisan on the Bayesian side.

From this answer, if I am not wrong, I conjecture that any task which suits a frequentist approach can be solved also in a Bayesian way, and vice versa.

I later learned that the rise of MCMC algorithms in the 80's lead to an exponential rise in the use of Bayes in methods as complex models built through hierarchical distributions suddenly were tractable(reference: Mixtures of Conjugate Priors and MCMC).

Then my question is that if the Bayesian approach is better than the frequentist (1. Jaynes taught me that in that book; 2. for instance more parameters in models like the hidden Markov model can learn more information and then be better to reach the reality, the posterior distribution than the Markov model; 3) more reasons from this answer to this question: What is the importance of probabilistic machine learning?) and it has been tractable, then why frequentist methods still dominate the machine learning field?

In the answers to this question I learned why frequentist is better practically. But I wonder if Bayesian can be as practical as frequentist approaches? How can it be?

Answer 1

MCMC is just an algorithm to sample from a distribution. On its own it is neither frequentist nor Bayesian. However, this algorithm has proven to be extremely useful for Bayesians. It has made it feasible to sample from the posterior distribution, and with those samples in hand, describe the posterior distribution. Before its use, real use of Bayesian statistics was mostly limited to models with conjugate priors. Meaning that before MCMC Bayesian statistics was more a theoretical, but unpractical, area of statistics. So most statistician would associate MCMC with Bayesian rather than frequentist statistics.

Is see both approaches as necessarily flawed approximations to answer to an unanswerable question: How do I give an answer when I don't have all the information? In the way I use statistics, the difference between the outcomes of these two methods is negligible. So I use pragmatic reasons to choose my method: unless Bayesian statistics has a clear added value, I will use frequentist because it is easier to "sell" my articles to journals. The potential added value of a Bayesian analysis would for example be if I have a prior that I really want to include in my analysis, and that prior is strong enough to influence my results. Especially, the latter is a fairly hard condition to meet in the type of analyses I do and the type of data I usually work with. So in the end I almost never use Bayesian statistics.

Some claim that the added value of the Bayesian is that the inference is more intuitive, and I kinda get their argument. However, it ignores the fact that most of my audience is not trained in Bayesian statistics. So until the majority is trained in Bayesian statistics, its potential cannot be realized. This is a catch 22: people are trained in frequentist statistics because that is what everybody uses, and everybody uses frequentists statistics because that is what everybody is trained in. However, we have to work with the world we live in, and not with the world we would want to live in.

Answer 2

The reasons for the scarce applications of Bayesian methods are two faceted: 1) additional layer of complexity; 2) lack of computationally cheap algorithms to handle Bayesian methods and big data, and people may don't keep up with the trends in this direction.

Some practical approaches:

1. MCMC is still expensive and hence variational inference approach is usually applied, for instance, MC dropout.
2. For the online predicting, usually we use an expensive Bayesian neural network as a teacher to distill its knowledge to a point estimate student for deployment.
3. Design better regularizations from the perspectives of Bayesian, for example Vadam.

References:

1. Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users