Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals

Question

A recent question on the difference between confidence and credible intervals led me to start re-reading Edwin Jaynes' article on that topic:

Jaynes, E. T., 1976. `Confidence Intervals vs Bayesian Intervals,' in Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science, W. L. Harper and C. A. Hooker (eds.), D. Reidel, Dordrecht, p. 175; (pdf)

In the abstract, Jaynes writes:

...we exhibit the Bayesian and orthodox solutions to six common statistical problems involving confidence intervals (including significance tests based on the same reasoning). In every case, we find the situation is exactly the opposite, i.e. the Bayesian method is easier to apply and yields the same or better results. Indeed, the orthodox results are satisfactory only when they agree closely (or exactly) with the Bayesian results. No contrary example has yet been produced.

(emphasis mine)

The paper was published in 1976, so perhaps things have moved on. My question is, are there examples where the frequentist confidence interval is clearly superior to the Bayesian credible interval (as per the challenge implicitly made by Jaynes)?

Examples based on incorrect prior assumptions are not acceptable as they say nothing about the internal consistency of the different approaches.

Answer 1

I said earlier that I would have a go at answering the question, so here goes...

Jaynes was being a little naughty in his paper in that a frequentist confidence interval isn't defined as an interval where we might expect the true value of the statistic to lie with high (specified) probability, so it isn't unduly surprising that contradictions arise if they are interpreted as if they were. The problem is that this is often the way confidence intervals are used in practice, as an interval highly likely to contain the true value (given what we can infer from our sample of data) is what we often want.

The key issue for me is that when a question is posed, it is best to have a direct answer to that question. Whether Bayesian credible intervals are worse than frequentist confidence intervals depends on what question was actually asked. If the question asked was:

(a) "Give me an interval where the true value of the statistic lies with probability p", then it appears a frequentist cannot actually answer that question directly (and this introduces the kind of problems that Jaynes discusses in his paper), but a Bayesian can, which is why a Bayesian credible interval is superior to the frequentist confidence interval in the examples given by Jaynes. But this is only becuase it is the "wrong question" for the frequentist.

(b) "Give me an interval where, were the experiment repeated a large number of times, the true value of the statistic would lie within p*100% of such intervals" then the frequentist answer is just what you want. The Bayesian may also be able to give a direct answer to this question (although it may not simply be the obvious credible interval). Whuber's comment on the question suggests this is the case.

So essentially, it is a matter of correctly specifying the question and properly intepreting the answer. If you want to ask question (a) then use a Bayesian credible interval, if you want to ask question (b) then use a frequentist confidence interval.

Answer 2

This is a "fleshed out" example given in a book written by Larry Wasserman All of statistics on Page 216 (12.8 Strengths and Weaknesses of Bayesian Inference). I basically provide what Wasserman doesn't in his book 1) an explanation for what is actually happening, rather than a throw away line; 2) the frequentist answer to the question, which Wasserman conveniently does not give; and 3) a demonstration that the equivalent confidence calculated using the same information suffers from the same problem.

In this example, he states the following situation

1. An observation, X, with a Sampling distribution: $(X|\theta)\sim N(\theta,1)$
2. Prior distribution of $(\theta)\sim N(0,1)$ (he actually uses a general $\tau^2$ for the variance, but his diagram specialises to $\tau^2=1$)

He then goes to show that, using a Bayesian 95% credible interval in this set-up eventually has 0% frequentist coverage when the true value of $\theta$ becomes arbitrarily large. For instance, he provides a graph of the coverage (p218), and checking by eye, when the true value of $\theta$ is 3, the coverage is about 35%. He then goes on to say:

...What should we conclude from all this? The important thing is to understand that frequentist and Bayesian methods are answering dierent questions. To combine prior beliefs with data in a principled way, use bayesian inference. To construct procedures with guaranteed long run performance, such as condence intervals, use frequentist methods... (p217)

And then moves on without any disection or explanation of why the Bayesian method performed apparently so bad. Further, he does not give a answer from the frequentist approach, just a broad brush statement about "the long-run" - a classical political tactic (emphasise your strength + others weakness, but never compare like for like).

I will show how the problem as stated $\tau=1$ can be formulated in frequentist/orthodox terms, and then show that the result using confidence intervals gives precisely the same answer as the Bayesian one. Thus any defect in the Bayesian (real or perceived) is not corrected by using confidence intervals.

Okay, so here goes. The first question I ask is what state of knowledge is described by the prior $\theta\sim N(0,1)$? If one was "ignorant" about $\theta$, then the appropriate way to express this is $p(\theta)\propto 1$. Now suppose that we were ignorant, and we observed $Y\sim N(\theta,1)$, independently of $X$. What would our posterior for $\theta$ be?

$$p(\theta|Y)\propto p(\theta)p(Y|\theta)\propto exp\Big(-\frac{1}{2}(Y-\theta)^2\Big)$$

Thus $(\theta|Y)\sim N(Y,1)$. This means that the prior distribution given in Wassermans example, is equivalent to having observed an iid copy of $X$ equal to $0$. Frequentist methods cannot deal with a prior, but it can be thought of as having made 2 observations from the sampling distribution, one equal to $0$, and one equal to $X$. Both problems are entirely equivalent, and we can actually give the frequentist answer for the question.

Because we are dealing with a normal distribution with known variance, the mean is a sufficient statistic for constructing a confidence interval for $\theta$. The mean is equal to $\overline{x}=\frac{0+X}{2}=\frac{X}{2}$ and has a sampling distribution

$$(\overline{x}|\theta)\sim N(\theta,\frac{1}{2})$$

Thus an $(1-\alpha)\text{%}$ CI is given by:

$$\frac{1}{2}X\pm Z_{\alpha/2}\frac{1}{\sqrt{2}}$$

But, using The results of example 12.8 for Wasserman, he shows that the posterior $(1-\alpha)\text{%}$ credible interval for $\theta$ is given by:

$$cX\pm \sqrt{c}Z_{\alpha/2}$$.

Where $c=\frac{\tau^{2}}{1+\tau^{2}}$. Thus, plugging in the value at $\tau^{2}=1$ gives $c=\frac{1}{2}$ and the credible interval becomes:

$$\frac{1}{2}X\pm Z_{\alpha/2}\frac{1}{\sqrt{2}}$$

Which are exactly the same as the confidence interval! So any defect in the coverage exhibited by the Bayesian method, is not corrected by using the frequentist confidence interval! [If the frequentist chooses to ignore the prior, then to be a fair comparison, the Bayesian should also ignore this prior, and use the ignorance prior $p(\theta)\propto 1$, and the two intervals will still be equal - both $X \pm Z_{\alpha/2})$].

So what the hell is going on here? The problem is basically one of non-robustness of the normal sampling distribution. because the problem is equivalent to having already observed a iid copy, $X=0$. If you have observed $0$, then this is extremely unlikely to have occurred if the true value is $\theta=4$ (probability that $X\leq 0$ when $\theta=4$ is 0.000032). This explains why the coverage is so bad for large "true values", because they effectively make the implicit observation contained in the prior an outlier. In fact you can show that this example is basically equivalent to showing that the arithmetic mean has an unbounded influence function.

Generalisation. Now some people may say "but you only considered $\tau=1$, which may be a special case". This is not true: any value of $\tau^2=\frac{1}{N}$ $(N=0,1,2,3,\dots)$ can be interpreted as observing $N$ iid copies of $X$ which were all equal to $0$, in addition to the $X$ of the question. The confidence interval will have the same "bad" coverage properties for large $\theta$. But this becomes increasingly unlikely if you keep observing values of $0$ (and no rational person would continue to worry about large $\theta$ when you keep seeing $0$).

Answer 3

Keith Winstein,

EDIT: Just to clarify, this answer describes the example given in Keith Winstein Answer on the King with the cruel statistical game. The Bayesian and Frequentist answers both use the same information, which is to ignore the information on the number of fair and unfair coins when constructing the intervals. If this information is not ignored, the the frequentist should use the integrated Beta-Binomial Likelihood as the sampling distribution in constructing the Confidence interval, in which case the Clopper-Pearson Confidence Interval is not appropriate, and needs to be modified. A similar adjustment should occur in the Bayesian solution.

EDIT: I have also clarified the initial use of the clopper Pearson Interval.

EDIT: alas, my alpha is the wrong way around, and my clopper pearson interval is incorrect. My humblest apologies to @whuber, who correctly pointed this out, but who I initially disagreed with and ignored.

The CI Using the Clopper Pearson method is very good

If you only get one observation, then the Clopper Pearson Interval can be evaluated analytically. Suppose the coin is comes up as "success" (heads) you need to choose $\theta$ such that

$$[Pr(Bi(1,\theta)\geq X)\geq\frac{\alpha}{2}] \cap [Pr(Bi(1,\theta)\leq X)\geq\frac{\alpha}{2}]$$

When $X=1$ these probabilities are $Pr(Bi(1,\theta)\geq 1)=\theta$ and $Pr(Bi(1,\theta)\leq 1)=1$, so the Clopper Pearson CI implies that $\theta\geq\frac{\alpha}{2}$ (and the trivially always true $1\geq\frac{\alpha}{2}$) when $X=1$. When $X=0$ these probabilities are $Pr(Bi(1,\theta)\geq 0)=1$ and $Pr(Bi(1,\theta)\leq 0)=1-\theta$, so the Clopper Pearson CI implies that $1-\theta \geq\frac{\alpha}{2}$, or $\theta\leq 1-\frac{\alpha}{2}$ when $X=0$. So for a 95% CI we get $[0.025,1]$ when $X=1$, and $[0,0.975]$ when $X=0$.

Thus, one who uses the Clopper Pearson Confidence Interval will never ever be beheaded. Upon observing the interval, it is basically the whole parameter space. But the C-P interval is doing this by giving 100% coverage to a supposedly 95% interval! Basically, the Frequentists "cheats" by giving a 95% confidence interval more coverage than he/she was asked to give (although who wouldn't cheat in such a situation? if it were me, I'd give the whole [0,1] interval). If the king asked for an exact 95% CI, this frequentist method would fail regardless of what actually happened (perhaps a better one exists?).

What about the Bayesian Interval? (specifically the Highest Posterior Desnity (HPD) Bayesian Interval)

Because we know a priori that both heads and tails can come up, the uniform prior is a reasonable choice. This gives a posterior distribution of $(\theta|X)\sim Beta(1+X,2-X)$ . Now, all we need to do now is create an interval with 95% posterior probability. Similar to the clopper pearson CI, the Cummulative Beta distribution is analytic here also, so that $Pr(\theta \geq \theta^{e} | x=1) = 1-(\theta^{e})^{2}$ and $Pr(\theta \leq \theta^{e} | x=0) = 1-(1-\theta^{e})^{2}$ setting these to 0.95 gives $\theta^{e}=\sqrt{0.05}\approx 0.224$ when $X=1$ and $\theta^{e}= 1-\sqrt{0.05}\approx 0.776$ when $X=0$. So the two credible intervals are $(0,0.776)$ when $X=0$ and $(0.224,1)$ when $X=1$

Thus the Bayesian will be beheaded for his HPD Credible interval in the case when he gets the bad coin and the Bad coin comes up tails which will occur with a chance of $\frac{1}{10^{12}+1}\times\frac{1}{10}\approx 0$.

First observation, the Bayesian Interval is smaller than the confidence interval. Another thing is that the Bayesian would be closer to the actual coverage stated, 95%, than the frequentist. In fact, the Bayesian is just about as close to the 95% coverage as one can get in this problem. And contrary to Keith's statement, if the bad coin is chosen, 10 Bayesians out of 100 will on average lose their head (not all of them, because the bad coin must come up heads for the interval to not contain $0.1$).

Interestingly, if the CP-interval for 1 observation was used repeatedly (so we have N such intervals, each based on 1 observation), and the true proportion was anything between $0.025$ and $0.975$, then coverage of the 95% CI will always be 100%, and not 95%! This clearly depends on the true value of the parameter! So this is at least one case where repeated use of a confidence interval does not lead to the desired level of confidence.

To quote a genuine 95% confidence interval, then by definition there should be some cases (i.e. at least one) of the observed interval which do not contain the true value of the parameter. Otherwise, how can one justify the 95% tag? Would it not be just a valid or invalid to call it a 90%, 50%, 20%, or even 0% interval?

I do not see how simply stating "it actually means 95% or more" without a complimentary restriction is satisfactory. This is because the obvious mathematical solution is the whole parameter space, and the problem is trivial. suppose I want a 50% CI? if it only bounds the false negatives then the whole parameter space is a valid CI using only this criteria.

Perhaps a better criterion is (and this is what I believe is implicit in the definition by Kieth) "as close to 95% as possible, without going below 95%". The Bayesian Interval would have a coverage closer to 95% than the frequentist (although not by much), and would not go under 95% in the coverage ($\text{100%}$ coverage when $X=0$, and $100\times\frac{10^{12}+\frac{9}{10}}{10^{12}+1}\text{%} > \text{95%}$ coverage when $X=1$).

In closing, it does seem a bit odd to ask for an interval of uncertainty, and then evaluate that interval by the using the true value which we were uncertain about. A "fairer" comparison, for both confidence and credible intervals, to me seems like the truth of the statement of uncertainty given with the interval.

Answer 4

The problem starts with your sentence :

Examples based on incorrect prior assumptions are not acceptable as they say nothing about the internal consistency of the different approaches.

Yeah well, how do you know your prior is correct?

Take the case of Bayesian inference in phylogeny. The probability of at least one change is related to evolutionary time (branch length t) by the formula

$$P=1-e^{-\frac{4}{3}ut}$$

with u being the rate of substitution.

Now you want to make a model of the evolution, based on comparison of DNA sequences. In essence, you try to estimate a tree in which you try to model the amount of change between the DNA sequences as close as possible. The P above is the chance of at least one change on a given branch. Evolutionary models describe the chances of change between any two nucleotides, and from these evolutionary models the estimation function is derived, either with p as a parameter or with t as a parameter.

You have no sensible knowledge and you chose a flat prior for p. This inherently implies an exponentially decreasing prior for t. (It becomes even more problematic if you want to set a flat prior on t. The implied prior on p is strongly dependent on where you cut off the range of t.)

In theory, t can be infinite, but when you allow an infinite range, the area under its density function equals infinity as well, so you have to define a truncation point for the prior. Now when you chose the truncation point sufficiently large, it is not difficult to prove that both ends of the credible interval rise, and at a certain point the true value is not contained in the credible interval any more. Unless you have a very good idea about the prior, Bayesian methods are not guaranteed to be equal to or superior to other methods.

ref: Joseph Felsenstein : Inferring Phylogenies, chapter 18

On a side note, I'm getting sick of that Bayesian/Frequentist quarrel. They're both different frameworks, and neither is the Absolute Truth. The classical examples pro Bayesian methods invariantly come from probability calculation, and not one frequentist will contradict them. The classical argument against Bayesian methods invariantly involve the arbitrary choice of a prior. And sensible priors are definitely possible.

It all boils down to the correct use of either method at the right time. I've seen very few arguments/comparisons where both methods were applied correctly. Assumptions of any method are very much underrated and far too often ignored.

EDIT : to clarify, the problem lies in the fact that the estimate based on p differs from the estimate based on t in the Bayesian framework when working with uninformative priors (which is in a number of cases the only possible solution). This is not true in the ML framework for phylogenetic inference. It is not a matter of a wrong prior, it is inherent to the method.

Answer 5

Frequentist confidence intervals bound the rate of false positives (Type I errors), and guarantee their coverage will be bounded below by the confidence parameter, even in the worst case. Bayesian credibility intervals don't.

So if the thing you care about is false positives and you need to bound them, confidence intervals are the the approach that you'll want to use.

For example, let's say you have an evil king with a court of 100 courtiers and courtesans and he wants to play a cruel statistical game with them. The king has a bag of a trillion fair coins, plus one unfair coin whose heads probability is 10%. He's going to perform the following game. First, he'll draw a coin uniformly at random from the bag.

Then the coin will be passed around a room of 100 people and each one will be forced to do an experiment on it, privately, and then each person will state a 95% uncertainty interval on what they think the coin's heads probability is.

Anybody who gives an interval that represents a false positive -- i.e. an interval that doesn't cover the true value of the heads probability -- will be beheaded.

If we wanted to express the /a posteriori/ probability distribution function of the coin's weight, then of course a credibility interval is what does that. The answer will always be the interval [0.5, 0.5] irrespective of outcome. Even if you flip zero heads or one head, you'll still say [0.5, 0.5] because it's a heck of a lot more probable that the king drew a fair coin and you had a 1/1024 day getting ten heads in a row, than that the king drew the unfair coin.

So this is not a good idea for the courtiers and courtesans to use! Because when the unfair coin is drawn, the whole room (all 100 people) will be wrong and they'll all get beheaded.

In this world where the most important thing is false positives, what we need is an absolute guarantee that the rate of false positives will be less than 5%, no matter which coin is drawn. Then we need to use a confidence interval, like Blyth-Still-Casella or Clopper-Pearson, that works and provides at least 95% coverage irrespective of the true value of the parameter, even in the worst case. If everybody uses this method instead, then no matter which coin is drawn, at the end of the day we can guarantee that the expected number of wrong people will be no more than five.

So the point is: if your criterion requires bounding false positives (or equivalently, guaranteeing coverage), you gotta go with a confidence interval. That's what they do. Credibility intervals may be a more intuitive way of expressing uncertainty, they may perform pretty well from a frequentist analysis, but they are not going to provide the guaranteed bound on false positives you'll get when you go asking for it.

(Of course if you also care about false negatives, you'll need a method that makes guarantees about those too...)

Answer 6

are there examples where the frequentist confidence interval is clearly superior to the Bayesian credible interval (as per the challenge implicitly made by Jaynes).

Here is an example: the true $\theta$ equals $10$ but the prior on $\theta$ is concentrated about $1$. I am doing statistics for a clinical trial, and $\theta$ measures the risk to death, so the Bayesian result is a disaster, isn't it ? More seriously, what is "the" Bayesian credible interval ? In other words: what is the selected prior ? Maybe Jaynes proposed an automatic way to select a prior, I don't know !

Bernardo proposed a "reference prior" to be used as a standard for scientific communication [and even a "reference credible interval" (Bernardo - objective credible regions)]. Assuming this is "the" Bayesian approach, now the question is: when is an interval superior to another one ? The frequentist properties of the Bayesian interval are not always optimal, but neither are the Bayesian properties of "the" frequentist interval
(by the way, what is "the" frequentist interval ? )