9
$\begingroup$

Recently, I have been reading about how Confidence Intervals in the Frequentist setting are often misinterpreted (e.g. The importance of a correct interpretation of a confidence interval).

Supposedly, many people incorrectly use the following understanding to incorrectly draw conclusions from Frequentist Confidence Intervals.

Given the following understanding of Confidence Intervals:

  • If you collect some data, fit a regression model to this data and estimate regression coefficients (e.g. b1) and then calculate the 95% confidence interval for b1 (e.g. between some range x,y)
  • It is incorrect to say "there is a 0.95 probability that this confidence interval contains the true value of b1.
  • However, if you could collect data many times, 95% of these times the original interval (x,y) would contain the true value of b1 (i.e. b1 from the population).

The following incorrect understanding is then derived by many people:

  • Someone could just turn the tables and say - that is a 0.95 probability that the data I collected and the confidence interval calculated from this data that I collected is one of those experiments in which the confidence interval bounds the true value of the parameter.

  • Furthermore, - many of these confidence intervals (e.g. https://i.stack.imgur.com/JwPFh.png) could contain the true value of the parameter AND many of these confidence intervals that contain the true value of the parameter have significant overlap with each other

  • Since most datasets would produce a confidence interval that would bound the true value of the parameter and most confidence intervals from all these datasets overlap-for both of these statements to be simultaneously true -it is not unreasonable to believe that in general, the CI calculated from the data I collected in general has a 0.95 prob of containing the true parameter.

This is being said, I have read that in the Bayesian setting, a similar concept called "Credible Intervals" are able to address some of the shortcomings present in "Confidence Intervals". Particularly, a Bayesian Credible Interval can be said to describe the probability of the true population parameter being contained in the resulting interval.

This brings me to my question:

  • I read that "Non Informative Priors" often result in the same parameter estimates as MLE (Maximum Likelihood Estimation), because "Non Informative Priors" are said to "let the data speak for itself" and have minimum influence on the parameter estimates. Non Informative Priors are also advantageous in the sense that if you genuinely do not have any strong "prior" beliefs in the distributions of the model parameters and do not want to incur any risk by incorrectly choosing some ill-suited distributions - Non Informative Priors will incur less risk compared to a "standard choice" in "priors" (i.e. Not-Non Informative Priors).

  • At the same time, Credible Intervals in the Bayesian setting seem to have more advantageous interpretations compared to Confidence Intervals in the Frequentist setting.

  • Thus, if both of these statements are true - why don't people just use "Non Informative Priors" to effectively perform Maximum Likelihood Estimation, while simultaneously deriving intervals on their estimates (i.e. Credible Intervals) with more advantageous interpretations? Would this not allow them to get the "best of both worlds"?

Currently, my guess is that this (i.e. using Non Informative Priors) is more complicated to perform (e.g. computation) and explain to an audience, thus it ends up being used less. Can someone please comment on this?

References:

$\endgroup$
6
  • 2
    $\begingroup$ Note that many Bayesians think that objective frequentist probability does not exist (as de Finetti wrote explicitly). If this is so, there is no such thing as a "true parameter", and Bayesian credible intervals can therefore not be interpreted in terms of "probabilities for where the true parameter is" either. $\endgroup$ Dec 25, 2022 at 10:51
  • 4
    $\begingroup$ Your title is about non-informative priors, your question is apparently about weakly informative priors. That's not the same thing. Which one are you interested in? $\endgroup$ Dec 25, 2022 at 10:53
  • 1
    $\begingroup$ In fact, from my reading of the Bayesian literature, supposedly non-informative priors are in fact rather popular (although one can argue about whether they are really non-informative). Weakly informative priors less so, I guess because they require choices that are often hard to make and justify, and then it's hard to tell what qualifies as "weak". $\endgroup$ Dec 25, 2022 at 10:58
  • 2
    $\begingroup$ Quite relevant: statmodeling.stat.columbia.edu/2015/05/01/… $\endgroup$ Dec 25, 2022 at 11:34
  • 1
    $\begingroup$ Looking at your 3 dotpoints in your incorrect understanding. In the first, sure someone could claim 95% or in fact any number between 0 and 1 as their probability that the parameter is in their confidence interval but why would that be something the rest of us would be interested in. In the second, whether or not different confidence intervals overlap is meaningless in this context. $\endgroup$ Dec 25, 2022 at 19:42

6 Answers 6

6
$\begingroup$

As mentioned by others in the comments, the Bayesian approach estimates slightly different things than the frequentist and the results have different interpretations. So even in cases where you could expect the results to be similar, they are not the same thing.

Start with the thread What is an "uninformative prior"? Can we ever have one with truly no information?. You would learn that there's no such thing as a “non-informative prior” as each prior brings some information. The Bayesian approach would give the same result as the maximum likelihood when using improper, flat prior $p(\theta) \propto 1$, and looking at the mode of the posterior, because it's like maximizing the likelihood alone

$$\begin{align} &\underset{\theta}{\operatorname{arg\,max}} \; p(X|\theta) \, p(\theta) = \\ &\underset{\theta}{\operatorname{arg\,max}} \; p(X|\theta) \times 1 \end{align}$$

But using flat priors has its own problems, like no guarantee of ending with the proper posterior distribution.

So the three main reasons are: ideological (different interpretations), that the priors do bring information to the model, and that using such priors leads to different complications. There's no free lunch.

$\endgroup$
3
  • $\begingroup$ @ Tim : thank you for your answer! How do we know that using a flat prior might sometimes result in an improper distribution? But in instances where a flat prior does result in a proper distribution - in such cases, will doing this give you the same results as mle with the added benefits of credible intervals (compared to confidence intervals)? $\endgroup$
    – stats_noob
    Dec 25, 2022 at 16:35
  • 1
    $\begingroup$ @stats_noob credible intervals and confidence intervals are different things. MLE is a point estimate that cannot be compared to a full posterior, unless you mean MAP. If you check questions tagged as bayesian you'd find such topics discussed many times. $\endgroup$
    – Tim
    Dec 25, 2022 at 17:28
  • $\begingroup$ I would recommend reading computer age statistical inference where they cover the differences between bayesian and frequentist - mainly chapter 3 $\endgroup$
    – seanv507
    Dec 27, 2022 at 8:45
11
$\begingroup$

This is a hard question because there are different Bayesian philosophies, different ideas of what a prior and posterior actually mean, and I'd even say there are also different versions of frequentism.

There are various issues here.

  1. Why do people still compute classical confidence intervals, and why isn't a Bayesian analysis more popular?

  2. How to interpret the results of the Bayesian analysis?

  3. Given that people do a Bayesian analysis, why aren't weakly informative priors more popular? (I don't think it makes sense to ask why non-informative priors aren't more popular given that the analysis is Bayesian, because I think they are really quite popular indeed, arguably too popular, see https://statmodeling.stat.columbia.edu/2015/05/01/general-problem-noninformatively-derived-bayesian-probabilities-tend-strong/ and the answer by @Tim.)

Ad (1): I believe that the requirement to specify a prior is a major issue that many have with a Bayesian approach. One reason for this can be convenience (thinking about a prior is additional work), another can be the idea that results are supposed to be objective rather than influenced by the subjective choice of a prior. A third reason is that, even if the requirement to specify a prior is accepted, in many situations existing information is of such a kind that it is very hard to translate this into a prior, and often there are various conceivable ways of doing this, and choosing one in particular is hard to justify. Note that I'm not saying that all these are good reasons, although I believe that particularly the last reason often makes a lot of sense, and in principle comprehensive sensitivity analysis would be required, exploring the implications of the choices of different priors that may all seem realistic.

Regarding subjectivity vs. objectivity, the thing is that there is also subjective impact in setting up a sampling model as frequentists do, and choices such as the confidence level. There is no way to determine these objectively from the data, and therefore the idea that only a Bayesian approach is affected by subjective choices is wrong. One may also argue that there are advantages in acknowledging necessary "non-objective" aspects of model choice rather than hiding them. For example here we argue that the ability to take into account multiple perspectives and context dependence is an advantage of an approach that requires non-objective input.

On the other hand, requiring additional subjective input (the Bayesian approach requires a prior on top of the other choices) isn't advantageous if it is unclear how to choose it and how to use it in an advantageous way. A prior helps if it is clear how the information encoded in the prior can improve the analysis; otherwise it is a much harder sell.

Ad (2): In the question it is stated that "credible Intervals in the Bayesian setting seem to have more advantageous interpretations compared to Confidence Intervals". I'm not so sure, and the interpretation of credible intervals depends on the specific school of Bayesian thought, and often it is ignored that there is more than one. For starters, many Bayesians believe that true frequentist distributions and true parameters do not exist, in which case an interpretation in terms of the probability that the "true parameter" is in the credible interval doesn't make much sense. There can be a long discussion about this, and some Bayesians may say that if they talk about a "true parameter" they mean something else than a true objectively existing frequentist parameter, but anyway, the Bayesian marketing claim that, as opposed to confidence intervals, "credible intervals give the users what they really want", namely probabilities regarding the true parameter, is highly problematic and not very convincing in my view.

If you indeed want posterior probabilities about true frequentist parameters, a Bayesian approach will have to be based on a frequentist probability concept for the sampling distribution, and it is "philosophically" difficult then to integrate this with a non-frequentist prior, at least as long as we're not in an "empirical Bayes" situation in which there is data generating process with repetition that can be interpreted convincingly as generating the parameters.

In any case, credible intervals and posterior probabilities in general are conditional on the specification of the prior, and if the prior is meaningless, so is the posterior. Therefore any prior choice needs meaningful justification and interpretation if the resulting posterior (and not, for example, only the resulting point estimator) is meant to be interpreted in terms of quantifying the "real" uncertainty. This applies to non-informative priors as well - one needs to argue why there is no information that allows a more precise choice, because otherwise the resulting quantification of uncertainty is not in line with what we actually know (which is the aim of Bayesian analysis in the first place).

Ad (3): Non-informative priors are actually quite popular because there are default choices (no subjective freedom!) and because users believe (in my view wrongly) that they do not need to put effort into the specification. If you choose a weakly informative prior, of course again you have to choose and justify how exactly to do it, and this makes them less popular. In fact default choices are controversial, and arguably any supposedly non-informative choice actually implicitly also encodes some information. So I don't think that they give "best of both worlds", rather the opposite, if anything (although there are situations in which they can be well motivated). The idea is that weakly informative priors encode a certain minimum amount of key information that people can easily agree on, but leave the data lots of power to determine the inference. This may often be reasonable, but doesn't solve all the problems either, see above.

$\endgroup$
3
$\begingroup$

As has been stated well by others the simplest answer to the original question is that "flat" priors are not realistic. Imagine doing a Bayesian power simulation relating to detecting an effect quantified by an odds ratio greater than 1.0, when the prior distribution of the log OR is flat. For the first simulated experiment we might draw an odds ratio of $10^5$ and get a Bayesian power of 1.0. Not interesting or realistic.

On a separate note, people frequently state that under "flat" priors Bayesian and frequentist methods are alike. This is true only in the very special case where there is a fixed sample size and one takes only one "look" at the data--at the end of the study at the planned sample size. So we need to downplay these types of comparisons (not to mention the Bayesian interpretation of the result being drastically different from the frequentist interpretation).

$\endgroup$
2
  • $\begingroup$ ""flat" priors are not realistic" it seems to me rather rare to have a problem where we genuinely have no prior knowledge at all. $\endgroup$ Dec 27, 2022 at 13:05
  • 1
    $\begingroup$ Exactly. For example in clinical trials we almost always know that the treatment is not a cure, so the odds ratio/hazard ratio/etc. cannot be 0 or infinity. $\endgroup$ Dec 27, 2022 at 14:28
2
$\begingroup$

I have this controversial view that the answer to this question is actually very simple and people over-complicate it by using academic responses. I also have asked the question, "why use frequentist methods when you can just get the same answer using Bayesian methods?", and I get academic responses which are not very convincing, e.g., "because the answer depends on the prior", ect. When I ran my own simulations and use absolutely horrendous priors the MCMC sampling still converged to the correct answer, which was almost the same as using non-informative priors, and which was almost the same as using frequentist methods. So the common response that, "since your answer depends on the prior", is not a what is seen in practice. So it is really a non-answer. The only time when the prior actually does matter is when you have small amounts of data, but then, statistical analysis derived from that data is highly suspicious anyway.

There are only three practical answers that make sense to me.

  • Frequentist methods, if applicable, are much faster than MCMC. For example, you can run least-squares regression calculators on gigantic datasets and get an answer extremely quickly. However, running Stan would be too slow. Also, related to this problem, sometimes setting up an MCMC simulation might cause some sort of bugs, and then you need to figure out what those are and why it is not running. With frequentist methods it is more reliable, but again, provided that the problem can be delt with in a frequentist manner.
  • Necessity is the mother of all invention. In the 1920s it was not possible to do Bayesian computation beyond the simplest problems. Therefore, the frequentist method was the default since it was the only way calculations could actually have been done. People are lazy. Once they learn how to do something one way, why would they then learn how to do it another way? For example, R is superior to SPSS, Stata, SAS, ect, so why do people not use R? Because they already can do it in using other software, so why put in the effort to learn something which is harder (even if better)?
  • Pedagogical reasons. Imagine incoming freshmen in Psychology are taking their first statistics course. You cannot talk about calculus, or MCMC algorithms, or MLE with gradient descent, ect. That will confuse everyone. Instead if you teach the students some commonly used sampling distributions, and how to look up their values, then they can actually learn how to do some statistical analysis. It is similar to asking the question, for an easy intro physics class, "why do they just not teach calculus and differential equations?". Because everyone would be confused, so they teach more elementary methods so the students can still get something out the class.
$\endgroup$
5
  • 2
    $\begingroup$ Hmm, so you are arguing that Fisher, Neyman etc rejected the Bayesian approach only because it was too hard to calculate it at the time. I see why you call your view controversial. $\endgroup$ Dec 27, 2022 at 11:12
  • 1
    $\begingroup$ "When I ran my own simulations and use absolutely horrendous priors the MCMC sampling still converged to the correct answer, which was almost the same as using non-informative priors, and which was almost the same as using frequentist methods." This will for sure depend on what kind of problems you're looking at. Not only really small sample sizes are a problem but also complex models with many (or only weakly identified) parameters. These are not rare these days. Also I suspect there will be trouble with misspecified models, which happen frequently as well. $\endgroup$ Dec 27, 2022 at 23:21
  • $\begingroup$ Also the problem of accurate quantification of uncertainty for a given sample size is different from "converging to the correct answer". $\endgroup$ Dec 27, 2022 at 23:23
  • $\begingroup$ @GrahamBornholt They rejected it for (emphasis on the word) academic reasons. Indeed, back in those days they did not have advanced computing to verify that the Bayesian answer is nearly identical. If they lived during this era then they might think differently. $\endgroup$ Dec 28, 2022 at 3:09
  • 1
    $\begingroup$ It is easy to forget that people like Fisher reject the Bayesian approach precisely because they are applied statisticians who don't find Bayesian interpretations fit for their purposes. They are not so much interested in quantifying their own uncertainty but instead try to understand the data generating process through the use of random variables and their associated (admittedly idealised) models. $\endgroup$ Dec 28, 2022 at 3:54
2
$\begingroup$

"Credible Intervals in the Bayesian setting seem to have more advantageous interpretations compared to Confidence Intervals in the Frequentist setting" - this is true only if the prior being used in the Bayesian setting is reflective of the true state of nature. If you use a "garbage" prior, you get garbage credible intervals. If you use a non-informatibe prior, you get an interval that is only credible if the non-informative prior is a reasonable prior for the problem at hand. To put it another way, non-informative priors are not used more often because there are not that many real-world problems where non-informative priors are reasonable for the problem at hand. To put it yet another way, there is no free lunch. By the way, if you really want a true confidence iterval, you can use a set-based minimax estimate under 0-1 loss.

$\endgroup$
2
  • 1
    $\begingroup$ There is no true state of nature and even if there were, the prior need not match it. The prior is the lens through which the evidence judge quantifies post-data beliefs in values of known quantities. $\endgroup$ Dec 27, 2022 at 10:25
  • 1
    $\begingroup$ "There is no true state of nature" sounds like a subjective belief ;) Perhaps @user was referring to the good frequentist properties that Bayesian methods have when the true parameter is generated according to the prior distribution. $\endgroup$ Dec 28, 2022 at 17:26
1
$\begingroup$

If you're not using informative priors, you're not being philosophically Bayesian. If the prior doesn't represent your uncertainty before seeing the data, then your posterior doesn't represent your uncertainty after seeing the data. So all interpretational advantages, such credible intervals vs confidence intervals, are lost.

In fact, you're being a frequentist ("these methods have good parameter coverage") but using tools that have traditionally been associated with Bayesian methods.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.