How is the bayesian framework better in interpretation when we usually use uninformative or subjective priors?

Question

It is often argued that the bayesian framework has a big advantage in interpretation (over frequentist), because it computes the probability of a parameter given the data - $p(\theta|x)$ instead of $p(x|\theta)$ as in the frequentist framework. So far so good.

But, the whole equation it is based on:

$p(\theta|x) = {p(x|\theta) . p(\theta) \over p(x)}$

looks to me little suspicious for 2 reasons:

1. In many papers, usualy uninformative priors (uniform distributions) are used and then just $p(\theta|x) = p(x|\theta)$, so bayesians get the same result as frequentists get - so how is then bayesian framework better in interpretation, when bayesian posterior and frequentists likelihood are the same distributions? It just yields the same result.

2. When using informative priors, you get different results, but the bayesian is affected by the subjective prior, so the whole $p(\theta|x)$ has the subjective tinge too.

In other words, the whole argument of $p(\theta|x)$ being better in interpretation than $p(x|\theta)$ builds on a presumption that $p(\theta)$ is kind of "real", which normally is not, it is just a starting point we somehow choose to make the MCMC run, a presumption, but it is not a description of reality (it can't be defined I think).

So how can we argue that bayesian is better in interpretation?

Answer 1

To give a more narrow response than the excellent ones that have already been posted, and focus on the advantage in interpretation - the Bayesian interpretation of a, e.g., "95% credible interval" is that the probability that the true parameter value lies within the interval equals 95%. One of the two common frequentist interpretations of a, e.g., "95% confidence interval", even if numerically the two are identical, is that in the long run, if we were to perform the procedure many many times, the frequency with which the interval would cover the real value would converge to 95%. The former is intuitive, the latter is not. Try explaining to a manager some time that you can't say "The probability that our solar panels will degrade by less than 20% over 25 years is 95%", but must instead say "If the true degradation rate was 20% over 25 years, and we could somehow repeat our sampling but with different results blah blah parallel identical universes blah, the long run frequency of times that the one-sided confidence interval I would calculate would lie entirely below 20%/25 years would be 5%", or whatever the equivalent frequentist statement would be.

An alternative frequentist interpretation would be "Before the data was generated, there was a 5% chance the interval I would calculate using the procedure I settled on would fall entirely below the true parameter value. However, now that we've collected the data, we can't make any such statement, because we're not subjectivists and the probability is either 0 or 1, depending upon whether it does or does not lie entirely below the true parameter value." That'll help with the auditors and when calculating a warranty reserve. (I actually find this definition reasonable, albeit not usually useful; it's also not easy to understand intuitively, and especially not if you're not a statistician.)

Neither frequentist interpretation is intuitive. The Bayesian version is. Hence the "big advantage in interpretation" held by the Bayesian approach.

Answer 2

In my opinion, the reason that Bayesian statistics are "better" for intepretation is nothing to do with the priors, but is due to the definition of a probability. The Bayesian definition (the relative plausibility of the truth of some proposition) is more closely in accord with our everyday usage of the word than is the frequentist definition (the long run frequency with which something occurrs). In most practical situations $p(\theta|x)$ is what we actually want to know, not $p(x|\theta)$, and the difficulty arises with frequentist statistics due to a tendency to interpret the results in a frequentist calculation as if it were a Bayesian one, i.e. $p(x|\theta)$ as if it were $p(\theta|x)$ (for example the p-value fallacy, or interpreting a confidence interval as if it were a credible interval).

Note that informative priors are not necessarily subjective, for instance I would not consider it subjective knowledge to assert that prior knowledge of some physical system should be independent of the units of measurement (as they are essentially arbitrary), leading to the idea of transformation groups and "minimally informative" priors.

The flip side of ignoring subjective knowledge is that your system may be sub-optimal because you are ignoring expert knowledge, so subjectivity is not necessarily a bad thing. For instance in the usual "infer the bias of a coin" problem, often used as a motivating example, you will learn relatively slowly with a uniform prior as the data comes in. But are all amounts of bias being equally likely a reasonable assumption? No, it is easy to make a slightly biased coin, or one that is completely biased (two heads or two tals), so if we build that assummption into our analysis, via a subjective prior, we will need less data to identify what the bias actually is.

Frequentist analyses also often contain subjective elements (for instance the decision to reject the null hypothesis if the p-value is less than 0.05, there is no logical compulsion to do so, it is merely a tradition that has proven useful). The advantage of the Bayesian approach is that the subjectivity is made explicit in the calculation, rather than leaving it implicit.

At the end of the day, it is a matter of "horses for courses", you should have both sets of tools in your toolbox, and be prepared to use the best tool for the task at hand.

Having said which, Bayesian $\gg$ frequentist !!! ;oP

Answer 3

The Bayesian framework has a big advantage over frequentist because it does not depend on having a "crystal ball" in terms of knowing the correct distributional assumptions to make. Bayesian methods depend on using what information you have, and knowing how to encode that information into a probability distribution.

Using Bayesian methods is basically using probability theory in its full power. Bayes theorem is nothing but a restatement of the classic product rule of probability theory:

$$p(\theta x|I)=p(\theta|I)p(x|\theta I)=p(x|I)p(\theta|xI)$$

So long as $p(x|I)\neq 0$ (i.e. the prior information didn't say what was observed was impossible) we can divide by it, and arrive at bayes theorm. I have used $I$ to denote the prior information, which is always present - you can't assign a probability distribution without information.

Now, if you think that Bayes theorem is suspect, then logically, you must also think that the product rule is also suspect. You can find a deductive argument here, which derives the product and sum rules, similar to Cox's theorem. A more explicit list of the assumptions required can be found here.

As far as I know, frequentist inference is not based on a set of foundations within a logical framework. Because it uses the Kolmogorov axioms of probability, there does not seem to be any connection between probability theory and statistical inference. There are not any axioms for frequentist inference which lead to a procedure that is to be followed. There are principles and methods (maximum likelihood, confidence intervals, p-values, etc.), and they work well, but they tend to be isolated and specialised to particular problems. I think frequentist methods are best left vague in their foundations, at least in terms of a strict logical framework.

For point $1$, getting the same result is somewhat irrelevant, from the perspective of interpretation. Two procedures may lead to the same result, but this need not mean that they are equivalent. If I was to just guess $\theta$, and happened to guess the maximum likelihood estimate (MLE), this would not mean that my guessing is just as good as MLE.

For point $2$, why should you be worried that people with different information will come to different conclusions? Someone with a phd in mathematics would, and should, come to different conclusions to someone with high school level mathematics. They have different amounts of information - why would we expect them to agree? When you are presented knew information, you tend to change your mind. How much depends on what kind of information it was. Bayes theorem contains this feature, as it should.

Using a uniform prior is often a convenient approximation to make when the likelihood is sharp compared to the prior. It is not worth the effort sometimes, to go through and properly set up a prior. Similarly, don't make the mistake of confusing Bayesian statistics with MCMC. MCMC is just an algorithm for integration, same as guassian quadratre, and in a similar class to the Laplace approximation. It is a bit more useful than quadratre because you can re-use the algorithm's output to do all your integrals (posterior means and variances are integrals), and a bit more general that Laplace because you don't need a big sample, or a well rounded peak in the posterior (Laplace is quicker though).

Answer 4

I have typically seen the uniform prior used in either "instructive" type examples, or in cases in which truly nothing is known about a particular hyperparameter. Typically, I see uninformed priors that provide little information about what the solution will be, but which encode mathematically what a good solution probably looks like. For example, one typically sees a Gaussian prior ($\mu=0$) placed over a regression coefficient, encoding the knowledge that all things being equal, we prefer solutions in which the coefficients have lower magnitudes. This is to avoid overfitting a data set, by finding solutions that do maximize the objective function but which don't make sense in the particular context of our problem. In a sense, they provide a way to give the statistical model some "clues" about a particular domain.

However, this isn't (in my opinion) the most important aspect of Bayesian methodologies. Bayesian methods are generative, in that they provide a complete "story" for how the data came into existence. Thus, they aren't simply pattern finders, but rather they are able to take into account the full reality of the situation at hand. For example, consider LDA (latent Dirichlet allocation), which provides a full generative story for how a text document comes to be, that goes something like this:

1. Select some mix of topics based on the likelihood of particular topics co-occurring; and
2. Select some set of words from the vocabulary, conditioned based on the selected topics.

Thus, the model is fit based on a very specific understanding of the objects in the domain (here, text documents) and how they got created; therefore, the information we get back is tailored directly to our problem domain (likelihoods of words given topics, likelihoods of topics being mentioned together, likelihoods of documents containing topics and to what extent, etc.). The fact that Bayes Theorem is required to do this is almost secondary, hence the little joke, "Bayes wouldn't be a Bayesian, and Christ wouldn't be a Christian."

In short, Bayesian models are all about rigorously modeling the domain objects using probability distributions; therefore, we are able to encode knowledge that wouldn't otherwise be available with a simple discriminative technique.