Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals

Question

A recent question on the difference between confidence and credible intervals led me to start re-reading Edwin Jaynes' article on that topic:

Jaynes, E. T., 1976. `Confidence Intervals vs Bayesian Intervals,' in Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science, W. L. Harper and C. A. Hooker (eds.), D. Reidel, Dordrecht, p. 175; (pdf)

In the abstract, Jaynes writes:

...we exhibit the Bayesian and orthodox solutions to six common statistical problems involving confidence intervals (including significance tests based on the same reasoning). In every case, we find the situation is exactly the opposite, i.e. the Bayesian method is easier to apply and yields the same or better results. Indeed, the orthodox results are satisfactory only when they agree closely (or exactly) with the Bayesian results. No contrary example has yet been produced.

(emphasis mine)

The paper was published in 1976, so perhaps things have moved on. My question is, are there examples where the frequentist confidence interval is clearly superior to the Bayesian credible interval (as per the challenge implicitly made by Jaynes)?

Examples based on incorrect prior assumptions are not acceptable as they say nothing about the internal consistency of the different approaches.

Answer 1

I said earlier that I would have a go at answering the question, so here goes...

Jaynes was being a little naughty in his paper in that a frequentist confidence interval isn't defined as an interval where we might expect the true value of the statistic to lie with high (specified) probability, so it isn't unduly surprising that contradictions arise if they are interpreted as if they were. The problem is that this is often the way confidence intervals are used in practice, as an interval highly likely to contain the true value (given what we can infer from our sample of data) is what we often want.

The key issue for me is that when a question is posed, it is best to have a direct answer to that question. Whether Bayesian credible intervals are worse than frequentist confidence intervals depends on what question was actually asked. If the question asked was:

(a) "Give me an interval where the true value of the statistic lies with probability p", then it appears a frequentist cannot actually answer that question directly (and this introduces the kind of problems that Jaynes discusses in his paper), but a Bayesian can, which is why a Bayesian credible interval is superior to the frequentist confidence interval in the examples given by Jaynes. But this is only becuase it is the "wrong question" for the frequentist.

(b) "Give me an interval where, were the experiment repeated a large number of times, the true value of the statistic would lie within p*100% of such intervals" then the frequentist answer is just what you want. The Bayesian may also be able to give a direct answer to this question (although it may not simply be the obvious credible interval). Whuber's comment on the question suggests this is the case.

So essentially, it is a matter of correctly specifying the question and properly intepreting the answer. If you want to ask question (a) then use a Bayesian credible interval, if you want to ask question (b) then use a frequentist confidence interval.

Answer 2

This is a "fleshed out" example given in a book written by Larry Wasserman All of statistics on Page 216 (12.8 Strengths and Weaknesses of Bayesian Inference). I basically provide what Wasserman doesn't in his book 1) an explanation for what is actually happening, rather than a throw away line; 2) the frequentist answer to the question, which Wasserman conveniently does not give; and 3) a demonstration that the equivalent confidence calculated using the same information suffers from the same problem.

In this example, he states the following situation

1. An observation, X, with a Sampling distribution: $(X|\theta)\sim N(\theta,1)$
2. Prior distribution of $(\theta)\sim N(0,1)$ (he actually uses a general $\tau^2$ for the variance, but his diagram specialises to $\tau^2=1$)

He then goes to show that, using a Bayesian 95% credible interval in this set-up eventually has 0% frequentist coverage when the true value of $\theta$ becomes arbitrarily large. For instance, he provides a graph of the coverage (p218), and checking by eye, when the true value of $\theta$ is 3, the coverage is about 35%. He then goes on to say:

...What should we conclude from all this? The important thing is to understand that frequentist and Bayesian methods are answering different questions. To combine prior beliefs with data in a principled way, use Bayesian inference. To construct procedures with guaranteed long run performance, such as confidence intervals, use frequentist methods... (p217)

And then moves on without any disection or explanation of why the Bayesian method performed apparently so bad. Further, he does not give a answer from the frequentist approach, just a broad brush statement about "the long-run" - a classical political tactic (emphasise your strength + others weakness, but never compare like for like).

I will show how the problem as stated $\tau=1$ can be formulated in frequentist/orthodox terms, and then show that the result using confidence intervals gives precisely the same answer as the Bayesian one. Thus any defect in the Bayesian (real or perceived) is not corrected by using confidence intervals.

Okay, so here goes. The first question I ask is what state of knowledge is described by the prior $\theta\sim N(0,1)$? If one was "ignorant" about $\theta$, then the appropriate way to express this is $p(\theta)\propto 1$. Now suppose that we were ignorant, and we observed $Y\sim N(\theta,1)$, independently of $X$. What would our posterior for $\theta$ be?

$$p(\theta|Y)\propto p(\theta)p(Y|\theta)\propto exp\Big(-\frac{1}{2}(Y-\theta)^2\Big)$$

Thus $(\theta|Y)\sim N(Y,1)$. This means that the prior distribution given in Wassermans example, is equivalent to having observed an iid copy of $X$ equal to $0$. Frequentist methods cannot deal with a prior, but it can be thought of as having made 2 observations from the sampling distribution, one equal to $0$, and one equal to $X$. Both problems are entirely equivalent, and we can actually give the frequentist answer for the question.

Because we are dealing with a normal distribution with known variance, the mean is a sufficient statistic for constructing a confidence interval for $\theta$. The mean is equal to $\overline{x}=\frac{0+X}{2}=\frac{X}{2}$ and has a sampling distribution

$$(\overline{x}|\theta)\sim N(\theta,\frac{1}{2})$$

Thus an $(1-\alpha)\text{%}$ CI is given by:

$$\frac{1}{2}X\pm Z_{\alpha/2}\frac{1}{\sqrt{2}}$$

But, using The results of example 12.8 for Wasserman, he shows that the posterior $(1-\alpha)\text{%}$ credible interval for $\theta$ is given by:

$$cX\pm \sqrt{c}Z_{\alpha/2}$$.

Where $c=\frac{\tau^{2}}{1+\tau^{2}}$. Thus, plugging in the value at $\tau^{2}=1$ gives $c=\frac{1}{2}$ and the credible interval becomes:

$$\frac{1}{2}X\pm Z_{\alpha/2}\frac{1}{\sqrt{2}}$$

Which are exactly the same as the confidence interval! So any defect in the coverage exhibited by the Bayesian method, is not corrected by using the frequentist confidence interval! [If the frequentist chooses to ignore the prior, then to be a fair comparison, the Bayesian should also ignore this prior, and use the ignorance prior $p(\theta)\propto 1$, and the two intervals will still be equal - both $X \pm Z_{\alpha/2})$].

So what the hell is going on here? The problem is basically one of non-robustness of the normal sampling distribution. because the problem is equivalent to having already observed a iid copy, $X=0$. If you have observed $0$, then this is extremely unlikely to have occurred if the true value is $\theta=4$ (probability that $X\leq 0$ when $\theta=4$ is 0.000032). This explains why the coverage is so bad for large "true values", because they effectively make the implicit observation contained in the prior an outlier. In fact you can show that this example is basically equivalent to showing that the arithmetic mean has an unbounded influence function.

Generalisation. Now some people may say "but you only considered $\tau=1$, which may be a special case". This is not true: any value of $\tau^2=\frac{1}{N}$ $(N=0,1,2,3,\dots)$ can be interpreted as observing $N$ iid copies of $X$ which were all equal to $0$, in addition to the $X$ of the question. The confidence interval will have the same "bad" coverage properties for large $\theta$. But this becomes increasingly unlikely if you keep observing values of $0$ (and no rational person would continue to worry about large $\theta$ when you keep seeing $0$).

Answer 3

The problem starts with your sentence :

Examples based on incorrect prior assumptions are not acceptable as they say nothing about the internal consistency of the different approaches.

Yeah well, how do you know your prior is correct?

Take the case of Bayesian inference in phylogeny. The probability of at least one change is related to evolutionary time (branch length t) by the formula

$$P=1-e^{-\frac{4}{3}ut}$$

with u being the rate of substitution.

Now you want to make a model of the evolution, based on comparison of DNA sequences. In essence, you try to estimate a tree in which you try to model the amount of change between the DNA sequences as close as possible. The P above is the chance of at least one change on a given branch. Evolutionary models describe the chances of change between any two nucleotides, and from these evolutionary models the estimation function is derived, either with p as a parameter or with t as a parameter.

You have no sensible knowledge and you chose a flat prior for p. This inherently implies an exponentially decreasing prior for t. (It becomes even more problematic if you want to set a flat prior on t. The implied prior on p is strongly dependent on where you cut off the range of t.)

In theory, t can be infinite, but when you allow an infinite range, the area under its density function equals infinity as well, so you have to define a truncation point for the prior. Now when you chose the truncation point sufficiently large, it is not difficult to prove that both ends of the credible interval rise, and at a certain point the true value is not contained in the credible interval any more. Unless you have a very good idea about the prior, Bayesian methods are not guaranteed to be equal to or superior to other methods.

ref: Joseph Felsenstein : Inferring Phylogenies, chapter 18

On a side note, I'm getting sick of that Bayesian/Frequentist quarrel. They're both different frameworks, and neither is the Absolute Truth. The classical examples pro Bayesian methods invariantly come from probability calculation, and not one frequentist will contradict them. The classical argument against Bayesian methods invariantly involve the arbitrary choice of a prior. And sensible priors are definitely possible.

It all boils down to the correct use of either method at the right time. I've seen very few arguments/comparisons where both methods were applied correctly. Assumptions of any method are very much underrated and far too often ignored.

EDIT : to clarify, the problem lies in the fact that the estimate based on p differs from the estimate based on t in the Bayesian framework when working with uninformative priors (which is in a number of cases the only possible solution). This is not true in the ML framework for phylogenetic inference. It is not a matter of a wrong prior, it is inherent to the method.

Answer 4

Keith Winstein,

EDIT: Just to clarify, this answer describes the example given in Keith Winstein Answer on the King with the cruel statistical game. The Bayesian and Frequentist answers both use the same information, which is to ignore the information on the number of fair and unfair coins when constructing the intervals. If this information is not ignored, the frequentist should use the integrated Beta-Binomial Likelihood as the sampling distribution in constructing the Confidence interval, in which case the Clopper-Pearson Confidence Interval is not appropriate, and needs to be modified. A similar adjustment should occur in the Bayesian solution.

EDIT: I have also clarified the initial use of the clopper Pearson Interval.

EDIT: alas, my alpha is the wrong way around, and my clopper pearson interval is incorrect. My humblest apologies to @whuber, who correctly pointed this out, but who I initially disagreed with and ignored.

The CI Using the Clopper Pearson method is very good

If you only get one observation, then the Clopper Pearson Interval can be evaluated analytically. Suppose the coin is comes up as "success" (heads) you need to choose $\theta$ such that

$$[Pr(Bi(1,\theta)\geq X)\geq\frac{\alpha}{2}] \cap [Pr(Bi(1,\theta)\leq X)\geq\frac{\alpha}{2}]$$

When $X=1$ these probabilities are $Pr(Bi(1,\theta)\geq 1)=\theta$ and $Pr(Bi(1,\theta)\leq 1)=1$, so the Clopper Pearson CI implies that $\theta\geq\frac{\alpha}{2}$ (and the trivially always true $1\geq\frac{\alpha}{2}$) when $X=1$. When $X=0$ these probabilities are $Pr(Bi(1,\theta)\geq 0)=1$ and $Pr(Bi(1,\theta)\leq 0)=1-\theta$, so the Clopper Pearson CI implies that $1-\theta \geq\frac{\alpha}{2}$, or $\theta\leq 1-\frac{\alpha}{2}$ when $X=0$. So for a 95% CI we get $[0.025,1]$ when $X=1$, and $[0,0.975]$ when $X=0$.

Thus, one who uses the Clopper Pearson Confidence Interval will never ever be beheaded. Upon observing the interval, it is basically the whole parameter space. But the C-P interval is doing this by giving 100% coverage to a supposedly 95% interval! Basically, the Frequentists "cheats" by giving a 95% confidence interval more coverage than he/she was asked to give (although who wouldn't cheat in such a situation? if it were me, I'd give the whole [0,1] interval). If the king asked for an exact 95% CI, this frequentist method would fail regardless of what actually happened (perhaps a better one exists?).

What about the Bayesian Interval? (specifically the Highest Posterior Desnity (HPD) Bayesian Interval)

Because we know a priori that both heads and tails can come up, the uniform prior is a reasonable choice. This gives a posterior distribution of $(\theta|X)\sim Beta(1+X,2-X)$ . Now, all we need to do now is create an interval with 95% posterior probability. Similar to the clopper pearson CI, the Cummulative Beta distribution is analytic here also, so that $Pr(\theta \geq \theta^{e} | x=1) = 1-(\theta^{e})^{2}$ and $Pr(\theta \leq \theta^{e} | x=0) = 1-(1-\theta^{e})^{2}$ setting these to 0.95 gives $\theta^{e}=\sqrt{0.05}\approx 0.224$ when $X=1$ and $\theta^{e}= 1-\sqrt{0.05}\approx 0.776$ when $X=0$. So the two credible intervals are $(0,0.776)$ when $X=0$ and $(0.224,1)$ when $X=1$

Thus the Bayesian will be beheaded for his HPD Credible interval in the case when he gets the bad coin and the Bad coin comes up tails which will occur with a chance of $\frac{1}{10^{12}+1}\times\frac{1}{10}\approx 0$.

First observation, the Bayesian Interval is smaller than the confidence interval. Another thing is that the Bayesian would be closer to the actual coverage stated, 95%, than the frequentist. In fact, the Bayesian is just about as close to the 95% coverage as one can get in this problem. And contrary to Keith's statement, if the bad coin is chosen, 10 Bayesians out of 100 will on average lose their head (not all of them, because the bad coin must come up heads for the interval to not contain $0.1$).

Interestingly, if the CP-interval for 1 observation was used repeatedly (so we have N such intervals, each based on 1 observation), and the true proportion was anything between $0.025$ and $0.975$, then coverage of the 95% CI will always be 100%, and not 95%! This clearly depends on the true value of the parameter! So this is at least one case where repeated use of a confidence interval does not lead to the desired level of confidence.

To quote a genuine 95% confidence interval, then by definition there should be some cases (i.e. at least one) of the observed interval which do not contain the true value of the parameter. Otherwise, how can one justify the 95% tag? Would it not be just a valid or invalid to call it a 90%, 50%, 20%, or even 0% interval?

I do not see how simply stating "it actually means 95% or more" without a complimentary restriction is satisfactory. This is because the obvious mathematical solution is the whole parameter space, and the problem is trivial. suppose I want a 50% CI? if it only bounds the false negatives then the whole parameter space is a valid CI using only this criteria.

Perhaps a better criterion is (and this is what I believe is implicit in the definition by Kieth) "as close to 95% as possible, without going below 95%". The Bayesian Interval would have a coverage closer to 95% than the frequentist (although not by much), and would not go under 95% in the coverage ($\text{100%}$ coverage when $X=0$, and $100\times\frac{10^{12}+\frac{9}{10}}{10^{12}+1}\text{%} > \text{95%}$ coverage when $X=1$).

In closing, it does seem a bit odd to ask for an interval of uncertainty, and then evaluate that interval by the using the true value which we were uncertain about. A "fairer" comparison, for both confidence and credible intervals, to me seems like the truth of the statement of uncertainty given with the interval.

Answer 5

In this answer I aim to describe the difference between confidence intervals and credible intervals in an intuitive way.

I hope that this may help to understand:

• why/how credible intervals are better than confidence intervals.
• on which conditions the credible interval depends and when they are not always better.

---

Credible intervals and confidence intervals are constructed in different ways and can be different

see also: The basic logic of constructing a confidence interval and If a credible interval has a flat prior, is a 95% confidence interval equal to a 95% credible interval?

In the question by probabilityislogic an example is given from Larry Wasserman, which was mentioned in the comments by suncoolsu.

$$X \sim N(\theta,1) \quad \text{where} \quad \theta \sim N(0,\tau^2)$$

We could see each experiment with random values for $\theta$ and $X$ as a joint variable. This is plotted below for the 20k simulated cases when $\tau=1$

This experiment can be considered as a joint random variable where both the observation $X$ and the underlying unobserved parameter $\theta$ have a multivariate normal distribution.

$$f(x,\theta) = \frac{1}{2 \pi \tau} e^{-\frac{1}{2} \left((x-\theta)^2+ \frac{1}{\tau^2}\theta^2\right)}$$

Both the $\alpha \%$-confidence interval and $\alpha \%$-credible interval draw boundaries in such a way that $\alpha \%$ of the mass of the density $f(\theta,X)$ falls inside the boundaries. How do they differ?

• The credible interval draws boundaries by evaluating the $\alpha \%$ mass in a horizontal direction such that for every fixed $X$ an $\alpha \%$ of the mass falls in between the boundaries for the conditional density $$\theta_X \sim N(cX,c) \quad \text{with} \quad c=\frac{\tau^2}{\tau^2+1}$$ falls in between the boundaries.

• The confidence interval draws boundaries by evaluating the $\alpha \%$ mass in a vertical direction such that for every fixed $\theta$ an $\alpha \%$ of the mass falls in between the boundaries for the conditional density $$X_\theta \sim N(\theta,1) \hphantom{ \quad \text{with} \quad c=\frac{\tau^2}{\tau^2+1}}$$

What is different?

The confidence interval is restricted in the way that it draws the boundaries. The confidence interval places these boundaries by considering the conditional distribution $X_\theta$ and will cover $\alpha \%$ independent from what the true value of $\theta$ is (this independence is both the strength and weakness of the confidence interval).

The credible interval makes an improvement by including information about the marginal distribution of $\theta$ and in this way it will be able to make smaller intervals without giving up on the average coverage which is still $\alpha \%$. (But it becomes less reliable/fails when the additional assumption, about the prior, is not true)

In the example the credible interval is smaller by a factor $c = \frac{\tau^2}{\tau^2+1}$ and the improvement of the coverage, albeit the smaller intervals, is achieved by shifting the intervals a bit towards $\theta = 0$, which has a larger probability of occurring (which is where the prior density concentrates).

Conclusion

We can say that*, if the assumptions are true then for a given observation $X$, the credible interval will always perform better (or at least the same). But yes, the exception is the disadvantage of the credible interval (and the advantage of the confidence interval) that the conditional cover probability $\alpha \%$ is biased depending on the true value of the parameter $\theta$. This is especially detrimental when the assumptions about the prior distribution of $\theta$ are not trustworthy.

*see also the two methods in this question The basic logic of constructing a confidence interval. In the image of my answer it is illustrated that the confidence interval can place the boundaries, with respect to the posterior distribution for a given observation $X$, at different 'heights'. So it may not always be optimally selecting the shortest interval, and for each observation $X$ it may be possible to decrease the length of the interval by shifting the boundaries while enclosing the same $\alpha \%$ amount of probability mass.

For a given underlying parameter $\theta$ the roles are reversed and it is the confidence interval that performs better (smaller interval in vertical direction) than the credible interval. (although this is not the performance that we seek because we are interested in the intervals in the other direction, intervals of $\theta$ given $X$ and not intervals of $X$ given $\theta$)

---

About the exception

Examples based on incorrect prior assumptions are not acceptable

This exclusion of incorrect assumptions makes it a bit a loaded question. Yes, given certain conditions, the credible interval is better than the confidence interval. But are those conditions practical?

Both credible intervals and confidence intervals make statements about some probability, like $\alpha \%$ of the cases the parameter is correctly estimated. However, that "probability" is only a probability in the mathematical sense and relates to the specific case that the underlying assumptions of the model are very trustworthy.

If the assumptions are uncertain then this uncertainty should propagate into the computed uncertainty/probability $\alpha \%$. So credible intervals and confidence intervals are in practice only appropriate when the assumptions are sufficiently trustworthy such that the propagation of errors can be neglected. Credible intervals might be in some cases easier to compute, but the additional assumptions, makes credible intervals (in some way) more difficult to apply than confidence intervals, because more assumptions are being made and this will influence the 'true' value of $\alpha \%$.

---

Additional:

This question relates a bit to Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?

See in the image below the expression of conditional probability/chance of containing the parameter for this particular example

The $\alpha \%$ confidence interval will correctly estimate/contain the true parameter $\alpha \%$ of the time, for a each parameter $\theta$. But for a given observation $X$ the $\alpha \%$ confidence interval will not estimate/contain the true parameter $\alpha \%$ of the time. (type I errors will occur at the same rate $\alpha \%$ for different values of the underlying parameter $\theta$. But for different observations $X$ the type I error rate will be different. For some observations the confidence interval may be more/less often wrong than for other observations).

The $\alpha \%$ credible interval will correctly estimate/contain the true parameter $\alpha \%$ of the time, for each observation $X$. But for a given parameter $\theta$ the $\alpha \%$ credible interval will not estimate/contain the true parameter $\alpha \%$ of the time. (type I errors will occur at the same rate $\alpha \%$ for different values of the observed parameter $X$. But for different underlying parameters $\theta$ the type I error rate will be different. For some underlying parameters the credible interval may be more/less often wrong than for other underlying parameters).

---

Code for computing both images:

# parameters
set.seed(1)
n <- 2*10^4
perc = 0.95
za <- qnorm(0.5+perc/2,0,1)

# model
tau <- 1
theta <- rnorm(n,0,tau)
X <- rnorm(n,theta,1)

# plot scatterdiagram of distribution
plot(theta,X, xlab=expression(theta), ylab = "observed X",
     pch=21,col=rgb(0,0,0,0.05),bg=rgb(0,0,0,0.05),cex=0.25,
     xlim = c(-5,5),ylim=c(-5,5)
    )

# confidence interval
t <- seq(-6,6,0.01)
lines(t,t-za*1,col=2)
lines(t,t+za*1,col=2)

# credible interval
obsX <- seq(-6,6,0.01)
lines(obsX*tau^2/(tau^2+1)+za*sqrt(tau^2/(tau^2+1)),obsX,col=3)
lines(obsX*tau^2/(tau^2+1)-za*sqrt(tau^2/(tau^2+1)),obsX,col=3)

# adding contours for joint density
conX <- seq(-5,5,0.1)
conT <- seq(-5,5,0.1)
ln <- length(conX)

z <- matrix(rep(0,ln^2),ln)
for (i in 1:ln) {
  for (j in 1:ln) {
    z[i,j] <- dnorm(conT[i],0,tau)*dnorm(conX[j],conT[i],1)
  }
}
contour(conT,conX,-log(z), add=TRUE, levels = 1:10 )

legend(-5,5,c("confidence interval","credible interval","log joint density"), lty=1, col=c(2,3,1), lwd=c(1,1,0.5),cex=0.7)
title(expression(atop("scatterplot and contourplot of", 
                      paste("X ~ N(",theta,",1)   and   ",theta," ~ N(0,",tau^2,")"))))




# expression succes rate as function of X and theta
# Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?
layout(matrix(c(1:2),1))
par(mar=c(4,4,2,2),mgp=c(2.5,1,0))
pX <- seq(-5,5,0.1)
pt <- seq(-5,5,0.1)
cc <- tau^2/(tau^2+1)

plot(-10,-10, xlim=c(-5,5),ylim = c(0,1),
     xlab = expression(theta), ylab = "chance of containing the parameter")
lines(pt,pnorm(pt/cc+za/sqrt(cc),pt,1)-pnorm(pt/cc-za/sqrt(cc),pt,1),col=3)
lines(pt,pnorm(pt+za,pt,1)-pnorm(pt-za,pt,1),col=2)
title(expression(paste("for different values ", theta)))

legend(-3.8,0.15,
       c("confidence interval","credible interval"),
       lty=1, col=c(2,3),cex=0.7, box.col="white")


plot(-10,-10, xlim=c(-5,5),ylim = c(0,1),
     xlab = expression(X), ylab = "chance of containing the parameter")
lines(pX,pnorm(pX*cc+za*sqrt(cc),pX*cc,sqrt(cc))-pnorm(pX*cc-za*sqrt(cc),pX*cc,sqrt(cc)),col=3)
lines(pX,pnorm(pX+za,pX*cc,sqrt(cc))-pnorm(pX-za,pX*cc,sqrt(cc)),col=2)
title(expression(paste("for different values ", X)))


text(0,0.3, 
     c("95% Confidence Interval\ndoes not imply\n95% chance of containing the parameter"),
     cex= 0.7,pos=1)

library(shape)
Arrows(-3,0.3,-3.9,0.38,arr.length=0.2)

Answer 6

Frequentist confidence intervals bound the rate of false positives (Type I errors), and guarantee their coverage will be bounded below by the confidence parameter, even in the worst case. Bayesian credibility intervals don't.

So if the thing you care about is false positives and you need to bound them, confidence intervals are the approach that you'll want to use.

For example, let's say you have an evil king with a court of 100 courtiers and courtesans and he wants to play a cruel statistical game with them. The king has a bag of a trillion fair coins, plus one unfair coin whose heads probability is 10%. He's going to perform the following game. First, he'll draw a coin uniformly at random from the bag.

Then the coin will be passed around a room of 100 people and each one will be forced to do an experiment on it, privately, and then each person will state a 95% uncertainty interval on what they think the coin's heads probability is.

Anybody who gives an interval that represents a false positive – i.e. an interval that doesn't cover the true value of the heads probability – will be beheaded.

If we wanted to express the a posteriori probability distribution function of the coin's weight, then of course a credibility interval is what does that. The answer will always be the interval [0.5, 0.5] irrespective of outcome. Even if you flip zero heads or one head, you'll still say [0.5, 0.5] because it's a heck of a lot more probable that the king drew a fair coin and you had a 1/1024 day getting ten heads in a row, than that the king drew the unfair coin.

So this is not a good idea for the courtiers and courtesans to use! Because when the unfair coin is drawn, the whole room (all 100 people) will be wrong and they'll all get beheaded.

In this world where the most important thing is false positives, what we need is an absolute guarantee that the rate of false positives will be less than 5%, no matter which coin is drawn. Then we need to use a confidence interval, like Blyth-Still-Casella or Clopper-Pearson, that works and provides at least 95% coverage irrespective of the true value of the parameter, even in the worst case. If everybody uses this method instead, then no matter which coin is drawn, at the end of the day we can guarantee that the expected number of wrong people will be no more than five.

So the point is: if your criterion requires bounding false positives (or equivalently, guaranteeing coverage), you gotta go with a confidence interval. That's what they do. Credibility intervals may be a more intuitive way of expressing uncertainty, they may perform pretty well from a frequentist analysis, but they are not going to provide the guaranteed bound on false positives you'll get when you go asking for it.

(Of course if you also care about false negatives, you'll need a method that makes guarantees about those too...)

Answer 7

Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals

I'm going to say "any paper in experimental science".

There's an XKCD cartoon that has made the rounds here before, which I've edited slightly:

Okay, the stick figure on the left is nuts, and the one on the right is saner. But I want to focus on a different question: if this experiment were published, what would you want to see in the paper?

You don't want the opinion of either of these guys. What you want is the information in the first panel, so you can form your own opinion. That's what the confidence interval tells you: the Universe—which we expect to lie to us about 5% of the time—just told us that the answer is somewhere in here.

That isn't what you really want to know. What you really want to know is something like the credible interval. But it's what you want the paper to tell you: it's a concise summary of the result of this particular experiment.

The calculation of the confidence interval still incorporates assumptions that may be wrong, invalidating it. But they're assumptions about the reliability of the equipment, the quality of the randomization, and other things that the experimenter can be expected to know better than you. Human bias can still creep in, but it's unavoidable that you have to trust the experimenter about these sorts of things.

If you want to make a decision on the basis of this data, then you shouldn't treat the confidence interval as a credible interval, as the guy on the left does. You probably should do a Bayesian analysis. Proponents of Bayesianism often talk about winning bets, because Bayesian inference is good for that. But not everything is about winning bets.

Answer 8

The second example in this thread compares a frequentist confidence interval to two different posterior intervals based on two different non-informative priors. Despite using all the information in the likelihood, both credible intervals can be considered inferior because: i) neither credible interval provides a long-run guarantee of covering the unknown fixed true parameter; ii) it is not obvious which non-informative prior one should choose when constructing the posterior if the experimenter truly has no prior knowledge; iii) the posterior probability statements are not verifiable statements about the actual fixed parameter, the hypothesis, nor the experiment.

Both the credible interval and the confidence interval attempt to address the request, "Give me a set of plausible true values of the parameter, given the observed data." In his answer to the original post, Dikran Marsupial provides the following:

(a) "Give me an interval where the true value of the statistic lies with probability p", then it appears a frequentist cannot actually answer that question directly (and this introduces the kind of problems that Jaynes discusses in his paper), but a Bayesian can, which is why a Bayesian credible interval is superior to the frequentist confidence interval in the examples given by Jaynes. But this is only becuase it is the "wrong question" for the frequentist.

(b) "Give me an interval where, were the experiment repeated a large number of times, the true value of the statistic would lie within p*100% of such intervals" then the frequentist answer is just what you want. The Bayesian may also be able to give a direct answer to this question (although it may not simply be the obvious credible interval). Whuber's comment on the question suggests this is the case.

Dikran Marsupial's response is wrong for two reasons. The first is that neither the credible interval nor the confidence interval is a set of statistic values. Each is a set in the parameter space. Secondly, if we ignore this mistake and consider both the confidence and credible interval as residing in the parameter space, it is misleading in (a) to suggest a Bayesian approach can provide "an interval where the true parameter lies with 100p% probability." Under a Bayesian approach it is more transparent to say "a set of values that has 100p% belief units (Bayesian probability)." We must make it clear this is not a verifiable statement about the actual fixed parameter, the hypothesis, nor the experiment. The confidence interval for a single observed experimental result is considered plausible due to its long-run performance over repeated experiments. This coverage probability is a statement about the experiment in relation to the unknown fixed true parameter. If the prior distribution is chosen in such a way that the posterior is dominated by the likelihood, Bayesian belief is more objectively viewed as a type of confidence based on frequency probability of the experiment.

Answer 9

are there examples where the frequentist confidence interval is clearly superior to the Bayesian credible interval (as per the challenge implicitly made by Jaynes).

Here is an example: the true $\theta$ equals $10$ but the prior on $\theta$ is concentrated about $1$. I am doing statistics for a clinical trial, and $\theta$ measures the risk to death, so the Bayesian result is a disaster, isn't it ? More seriously, what is "the" Bayesian credible interval ? In other words: what is the selected prior ? Maybe Jaynes proposed an automatic way to select a prior, I don't know !

Bernardo proposed a "reference prior" to be used as a standard for scientific communication [and even a "reference credible interval" (Bernardo - objective credible regions)]. Assuming this is "the" Bayesian approach, now the question is: when is an interval superior to another one ? The frequentist properties of the Bayesian interval are not always optimal, but neither are the Bayesian properties of "the" frequentist interval
(by the way, what is "the" frequentist interval ? )