Why Aren't "Non-Informative Priors" More Popular?

Question

Recently, I have been reading about how Confidence Intervals in the Frequentist setting are often misinterpreted (e.g. The importance of a correct interpretation of a confidence interval).

Supposedly, many people incorrectly use the following understanding to incorrectly draw conclusions from Frequentist Confidence Intervals.

Given the following understanding of Confidence Intervals:

• If you collect some data, fit a regression model to this data and estimate regression coefficients (e.g. b1) and then calculate the 95% confidence interval for b1 (e.g. between some range x,y)
• It is incorrect to say "there is a 0.95 probability that this confidence interval contains the true value of b1.
• However, if you could collect data many times, 95% of these times the original interval (x,y) would contain the true value of b1 (i.e. b1 from the population).

The following incorrect understanding is then derived by many people:

• Someone could just turn the tables and say - that is a 0.95 probability that the data I collected and the confidence interval calculated from this data that I collected is one of those experiments in which the confidence interval bounds the true value of the parameter.

• Furthermore, - many of these confidence intervals (e.g. https://i.stack.imgur.com/JwPFh.png) could contain the true value of the parameter AND many of these confidence intervals that contain the true value of the parameter have significant overlap with each other

• Since most datasets would produce a confidence interval that would bound the true value of the parameter and most confidence intervals from all these datasets overlap-for both of these statements to be simultaneously true -it is not unreasonable to believe that in general, the CI calculated from the data I collected in general has a 0.95 prob of containing the true parameter.

This is being said, I have read that in the Bayesian setting, a similar concept called "Credible Intervals" are able to address some of the shortcomings present in "Confidence Intervals". Particularly, a Bayesian Credible Interval can be said to describe the probability of the true population parameter being contained in the resulting interval.

This brings me to my question:

• I read that "Non Informative Priors" often result in the same parameter estimates as MLE (Maximum Likelihood Estimation), because "Non Informative Priors" are said to "let the data speak for itself" and have minimum influence on the parameter estimates. Non Informative Priors are also advantageous in the sense that if you genuinely do not have any strong "prior" beliefs in the distributions of the model parameters and do not want to incur any risk by incorrectly choosing some ill-suited distributions - Non Informative Priors will incur less risk compared to a "standard choice" in "priors" (i.e. Not-Non Informative Priors).

• At the same time, Credible Intervals in the Bayesian setting seem to have more advantageous interpretations compared to Confidence Intervals in the Frequentist setting.

• Thus, if both of these statements are true - why don't people just use "Non Informative Priors" to effectively perform Maximum Likelihood Estimation, while simultaneously deriving intervals on their estimates (i.e. Credible Intervals) with more advantageous interpretations? Would this not allow them to get the "best of both worlds"?

Currently, my guess is that this (i.e. using Non Informative Priors) is more complicated to perform (e.g. computation) and explain to an audience, thus it ends up being used less. Can someone please comment on this?

References:

• Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?
• From a Bayesian probability perspective, why doesn't a 95% confidence interval contain the true parameter with 95% probability?

Answer 1

As mentioned by others in the comments, the Bayesian approach estimates slightly different things than the frequentist and the results have different interpretations. So even in cases where you could expect the results to be similar, they are not the same thing.

Start with the thread What is an "uninformative prior"? Can we ever have one with truly no information?. You would learn that there's no such thing as a “non-informative prior” as each prior brings some information. The Bayesian approach would give the same result as the maximum likelihood when using improper, flat prior $p(\theta) \propto 1$, and looking at the mode of the posterior, because it's like maximizing the likelihood alone

$$\begin{align} &\underset{\theta}{\operatorname{arg\,max}} \; p(X|\theta) \, p(\theta) = \\ &\underset{\theta}{\operatorname{arg\,max}} \; p(X|\theta) \times 1 \end{align}$$

But using flat priors has its own problems, like no guarantee of ending with the proper posterior distribution.

So the three main reasons are: ideological (different interpretations), that the priors do bring information to the model, and that using such priors leads to different complications. There's no free lunch.

Answer 2

This is a hard question because there are different Bayesian philosophies, different ideas of what a prior and posterior actually mean, and I'd even say there are also different versions of frequentism.

There are various issues here.

1. Why do people still compute classical confidence intervals, and why isn't a Bayesian analysis more popular?

2. How to interpret the results of the Bayesian analysis?

3. Given that people do a Bayesian analysis, why aren't weakly informative priors more popular? (I don't think it makes sense to ask why non-informative priors aren't more popular given that the analysis is Bayesian, because I think they are really quite popular indeed, arguably too popular, see https://statmodeling.stat.columbia.edu/2015/05/01/general-problem-noninformatively-derived-bayesian-probabilities-tend-strong/ and the answer by @Tim.)

Ad (1): I believe that the requirement to specify a prior is a major issue that many have with a Bayesian approach. One reason for this can be convenience (thinking about a prior is additional work), another can be the idea that results are supposed to be objective rather than influenced by the subjective choice of a prior. A third reason is that, even if the requirement to specify a prior is accepted, in many situations existing information is of such a kind that it is very hard to translate this into a prior, and often there are various conceivable ways of doing this, and choosing one in particular is hard to justify. Note that I'm not saying that all these are good reasons, although I believe that particularly the last reason often makes a lot of sense, and in principle comprehensive sensitivity analysis would be required, exploring the implications of the choices of different priors that may all seem realistic.

Regarding subjectivity vs. objectivity, the thing is that there is also subjective impact in setting up a sampling model as frequentists do, and choices such as the confidence level. There is no way to determine these objectively from the data, and therefore the idea that only a Bayesian approach is affected by subjective choices is wrong. One may also argue that there are advantages in acknowledging necessary "non-objective" aspects of model choice rather than hiding them. For example here we argue that the ability to take into account multiple perspectives and context dependence is an advantage of an approach that requires non-objective input.

On the other hand, requiring additional subjective input (the Bayesian approach requires a prior on top of the other choices) isn't advantageous if it is unclear how to choose it and how to use it in an advantageous way. A prior helps if it is clear how the information encoded in the prior can improve the analysis; otherwise it is a much harder sell.

Ad (2): In the question it is stated that "credible Intervals in the Bayesian setting seem to have more advantageous interpretations compared to Confidence Intervals". I'm not so sure, and the interpretation of credible intervals depends on the specific school of Bayesian thought, and often it is ignored that there is more than one. For starters, many Bayesians believe that true frequentist distributions and true parameters do not exist, in which case an interpretation in terms of the probability that the "true parameter" is in the credible interval doesn't make much sense. There can be a long discussion about this, and some Bayesians may say that if they talk about a "true parameter" they mean something else than a true objectively existing frequentist parameter, but anyway, the Bayesian marketing claim that, as opposed to confidence intervals, "credible intervals give the users what they really want", namely probabilities regarding the true parameter, is highly problematic and not very convincing in my view.

If you indeed want posterior probabilities about true frequentist parameters, a Bayesian approach will have to be based on a frequentist probability concept for the sampling distribution, and it is "philosophically" difficult then to integrate this with a non-frequentist prior, at least as long as we're not in an "empirical Bayes" situation in which there is data generating process with repetition that can be interpreted convincingly as generating the parameters.

In any case, credible intervals and posterior probabilities in general are conditional on the specification of the prior, and if the prior is meaningless, so is the posterior. Therefore any prior choice needs meaningful justification and interpretation if the resulting posterior (and not, for example, only the resulting point estimator) is meant to be interpreted in terms of quantifying the "real" uncertainty. This applies to non-informative priors as well - one needs to argue why there is no information that allows a more precise choice, because otherwise the resulting quantification of uncertainty is not in line with what we actually know (which is the aim of Bayesian analysis in the first place).

Ad (3): Non-informative priors are actually quite popular because there are default choices (no subjective freedom!) and because users believe (in my view wrongly) that they do not need to put effort into the specification. If you choose a weakly informative prior, of course again you have to choose and justify how exactly to do it, and this makes them less popular. In fact default choices are controversial, and arguably any supposedly non-informative choice actually implicitly also encodes some information. So I don't think that they give "best of both worlds", rather the opposite, if anything (although there are situations in which they can be well motivated). The idea is that weakly informative priors encode a certain minimum amount of key information that people can easily agree on, but leave the data lots of power to determine the inference. This may often be reasonable, but doesn't solve all the problems either, see above.

Answer 3

As has been stated well by others the simplest answer to the original question is that "flat" priors are not realistic. Imagine doing a Bayesian power simulation relating to detecting an effect quantified by an odds ratio greater than 1.0, when the prior distribution of the log OR is flat. For the first simulated experiment we might draw an odds ratio of $10^5$ and get a Bayesian power of 1.0. Not interesting or realistic.

On a separate note, people frequently state that under "flat" priors Bayesian and frequentist methods are alike. This is true only in the very special case where there is a fixed sample size and one takes only one "look" at the data--at the end of the study at the planned sample size. So we need to downplay these types of comparisons (not to mention the Bayesian interpretation of the result being drastically different from the frequentist interpretation).

Answer 4

I have this controversial view that the answer to this question is actually very simple and people over-complicate it by using academic responses. I also have asked the question, "why use frequentist methods when you can just get the same answer using Bayesian methods?", and I get academic responses which are not very convincing, e.g., "because the answer depends on the prior", ect. When I ran my own simulations and use absolutely horrendous priors the MCMC sampling still converged to the correct answer, which was almost the same as using non-informative priors, and which was almost the same as using frequentist methods. So the common response that, "since your answer depends on the prior", is not a what is seen in practice. So it is really a non-answer. The only time when the prior actually does matter is when you have small amounts of data, but then, statistical analysis derived from that data is highly suspicious anyway.

There are only three practical answers that make sense to me.

• Frequentist methods, if applicable, are much faster than MCMC. For example, you can run least-squares regression calculators on gigantic datasets and get an answer extremely quickly. However, running Stan would be too slow. Also, related to this problem, sometimes setting up an MCMC simulation might cause some sort of bugs, and then you need to figure out what those are and why it is not running. With frequentist methods it is more reliable, but again, provided that the problem can be delt with in a frequentist manner.
• Necessity is the mother of all invention. In the 1920s it was not possible to do Bayesian computation beyond the simplest problems. Therefore, the frequentist method was the default since it was the only way calculations could actually have been done. People are lazy. Once they learn how to do something one way, why would they then learn how to do it another way? For example, R is superior to SPSS, Stata, SAS, ect, so why do people not use R? Because they already can do it in using other software, so why put in the effort to learn something which is harder (even if better)?
• Pedagogical reasons. Imagine incoming freshmen in Psychology are taking their first statistics course. You cannot talk about calculus, or MCMC algorithms, or MLE with gradient descent, ect. That will confuse everyone. Instead if you teach the students some commonly used sampling distributions, and how to look up their values, then they can actually learn how to do some statistical analysis. It is similar to asking the question, for an easy intro physics class, "why do they just not teach calculus and differential equations?". Because everyone would be confused, so they teach more elementary methods so the students can still get something out the class.

Answer 5

"Credible Intervals in the Bayesian setting seem to have more advantageous interpretations compared to Confidence Intervals in the Frequentist setting" - this is true only if the prior being used in the Bayesian setting is reflective of the true state of nature. If you use a "garbage" prior, you get garbage credible intervals. If you use a non-informatibe prior, you get an interval that is only credible if the non-informative prior is a reasonable prior for the problem at hand. To put it another way, non-informative priors are not used more often because there are not that many real-world problems where non-informative priors are reasonable for the problem at hand. To put it yet another way, there is no free lunch. By the way, if you really want a true confidence iterval, you can use a set-based minimax estimate under 0-1 loss.

Answer 6

If you're not using informative priors, you're not being philosophically Bayesian. If the prior doesn't represent your uncertainty before seeing the data, then your posterior doesn't represent your uncertainty after seeing the data. So all interpretational advantages, such credible intervals vs confidence intervals, are lost.

In fact, you're being a frequentist ("these methods have good parameter coverage") but using tools that have traditionally been associated with Bayesian methods.