Why I should use Bayesian inference with uninformative prior? [duplicate]

Question

I am a Ph.D. student and currently I am studying Bayesian inference concerning vector autoregressive models. A lot of researchers when talking about uninformative prior, conclude that the results of inference are equal to what we can obtain using OLS. My question is: if this is true, why I should use Bayesian inference instead of OLS?

Answer 1

First of all, there is no such a thing as "uninformative priors" (we rather talk about "weakly informative" priors), each prior brings some kind of assumptions into your model. On another hand, the more information does your data provides, the less influential your prior becomes.

But taking this aside, with "uninformative" priors the point estimates from your model are the same as if you used maximum likelihood estimation (see this discussed for linear regression). So why would we use Bayesian estimation with uninformative priors? Well, if you are interested only in point estimates, then it is basically the same. However with Bayesian estimation what you get is a posterior distribution of the parameters, so much more information than using maximum likelihood. This is the most elementary difference. There are more differences, but they all follow from the fact what we have a full probabilistic model and we would need a whole book to discuss all of them.

Answer 2

I will piggy-back on @Tim's answer, but with a little more detail.

As Tim stated, with a flat prior, your point estimates will be very similar. If your Bayesian point estimate is the maximum a posterior (MAP) estimate, it will always be exactly the same as using the MLE by definition. If your posterior distribution is symmetric and unimodal, then the posterior mean will also be exactly the as the MLE estimate. Furthermore, if the posterior distribution is Gaussian, then using the MLE for point estimates will give you the same point estimates as the MLE and using the Fisher's information to describe the covariance will give you back the exact same distribution as the posterior distribution.

In fact, MLE theory tells us that under certain conditions, the log likelihood (or log posterior density with a flat prior) will asymptotically approach a quadratic function. If the log posterior distribution is quadratic, this turns out to be exactly the same as the posterior being Gaussian. In otherwords, asymptotically, using the MLE for inference is exactly the same as using the full posterior distribution with a flat prior.

What this implies is that if you are going to use a flat prior, you should only consider using Bayesian MCMC methods if you are worried that asymptotic approximations are not precise enough. When is this a problem? I'm sure there's plenty of other examples that don't fit this rule, but I would say the most common case you should be concerned with is the case when the number of samples per parameter fit is low (recall that one of the necessary assumptions for MLE asymptotics is that the number of parameters fit divided by observations goes to 0). For example, consider mixed effects models. In these cases, we have at least one parameter per subject (although many observations per subject). In this case, it is common practice to use ReML rather than MLE estimates to obtain the variance estimates, as the MLE variance estimates are well known to be downward biased, and problematically so when you have so many parameters to estimate. However, using a Bayesian approach, there's no such issue: just sample from the posterior and don't worry about it!

Answer 3

There are several reasons to use a Bayesian method rather than a Frequentist or Likelihoodist method. This is even more true for vector autoregression.

First, let me begin with a trivial example of what appear to be "no differences" under a trivial and simple problem. The example comes from Thomas Bayes' original article that solved Bayes theorem.

His method involved a billiards table, but let's simplify it to computerized random number generation. His example is nicer in that the physics, the Frequentist interpretation, and the Bayesian interpretation are obviously linked.

This is a two player game. The first step is to generate a random number between 0 and 1, denoted $\theta.$ Random numbers will be generated between zero and one for each round of the game. If the random number is less than or equal to $\theta$ then the first player scores, otherwise the second player scores. The first player to six points wins.

Now imagine a score of 5-3, what are the odds that player two will win? It is here that Pearson-Neyman Frequentist, Fisherian Likelihoodist and Bayesian methods diverge.

For the Frequentist and the Likelihoodist, the estimated probability of winning any one round for player two is $3/8^{ths}$. The probability of player two winning three rounds in a row is $$\frac{3}{8}^3.$$ This is approximately 18:1 odds against player two.

For the Bayesian the question is different. First, the posterior probability has to be solved, which is a distribution and not a point estimate. With a flat prior where $p(\theta)\propto{1}$, the posterior probability under a binomial likelihood is $504\theta^3(1-\theta)^5$. Because this is a distribution, we must average over the entire posterior to make a prediction and eliminate the uncertainty regarding $\theta.$ This is solved by calculating $$\int_0^1504\theta^3\theta^3(1-\theta)^5.$$ The extra $\theta^3$ is the probability of winning three in a row, while the rest is the posterior. The resulting calculated odds are 10:1 against the second player winning. That is not trivially different from the Frequentist or Likelihoodist odds. Further, a bookie using null hypothesis methods could be "Dutch booked," or in simpler terms, a gambler or set of gamblers could construct a convex combination of gambles that would create a sure win for themselves due to the calculation differences.

A Bayesian prediction does not automatically have the same value as null hypothesis predictions. The non-Bayesian method can never stochastically dominate the Bayesian method, but the Bayesian method can stochastically dominate the null hypothesis method.

Which should you use? It depends on what you are doing with the prediction. If you are testing if some monetary regime impacts some measure of output, then you should use a null hypothesis method. If you are going to base a budget or gamble in some manner on the outcome, then you should use the Bayesian method. Bayesian solutions are inherently coherent and admissible, that is to say, that you can gamble on them and that there is no less risky way to construct an estimate.

While it is true that if your true model has normally distributed data, then under a flat prior you will get equivalent results when there are three or fewer independent variables, this will not be true for three or more in regression. This is due to Stein's paradox. In that case, you can always construct a Bayesian model that will be superior to a null hypothesis method, though you cannot use a "flat" prior as it will not integrate to unity.

Finally, for individuals in macroeconomics and in the capital markets, non-Bayesian methods are not valid solutions. Up until now, everyone has just assumed the distribution of the underlying data into existence. It was usually normal or lognormal. I have written a paper deriving the distribution of the underlying data. I show that there is no sufficient statistic and that least squares creates serious estimation errors. For the capital markets, it overestimates returns by two percent per annum and underestimates risk by four percent per annum.

To understand the magnitude of that bias, had a prediction been made one hundred years ago on the current per capita income of people under the British Raj, the error would have put Indian per capita income between Spain and Portugal's. India would be the largest economy in the world.

Although the papers are on equity securities, primarily, it applies to any model that has capital in it. You can find those papers at https://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=1541471 .

Answer 4

In addition to the great answers already posted, before you decide which method to use, you must first decide what you want to know. This is important because frequentist and Bayesian methods answer different questions.

Frequentists are worried about the statistical properties of the process which they use to calculate, in your example, a confidence interval. They don't particularly care if the specific confidence interval you calculated from your data contains your true parameter. Rather, they care that, after a million confidence intervals are calculated using a certain process, a certain proportion of them is guaranteed to contain the true parameter.

Bayesians, on the other hand, care about what information can be extracted from your data. They want to know how wide an interval must be, given your specific data, in order to contain a certain parameter with some pre-selected probability.

So if you want to quantify the statistical properties of a certain process for analysing data, then frequentist methods are the way. If, on the other hand, you want to know what your specific data is telling you, then you should use bayesian methods.

The fact that, under some very specific conditions, they both offer numerically similar answers shouldn't take away from the fact that they are answering different questions.