Should I have "Confidence" in Credibility Intervals?

Question

Preliminaries

First, I know that the Bayesian/Frequentist debate is rather long in tooth at this point, but I hope my question is sufficiently different from the others I reviewed on this site before I asked this question.

Lest others start forwarding me the numerous links on this site for "What is Bayesian Probability" or "Confidence Intervals vs Credibility Intervals", let me say that I am not concerned with the differences between Bayesian and Frequentist probabilities, as I understand that. What I want to know is why I should have any trust in Bayesian Credibility Intervals as a reliable description of the plausible range of parameter values.

My main question, then some background:

On what basis do we judge the reliability (or trustworthiness) of Bayesian Credibility Intervals?

I ask this because there is a conundrum with Credibility Intervals that I don't see with Confidence Intervals: how do I determine the 'risk' of being wrong?

An Example from Confidence Intervals

A 95% confidence interval is constructed to be right (i.e., cover the true parameter) in 95% of samples to which it is applied: the 95% is a probability that applies to the procedure over many samples. However, just like a realization of a random variable doesn't have to be anywhere near its mean value, any individual CI does not, in principle, have to bracket anything even close to the true value.

But...when we perform inference, we form a CI in an attempt to show the plausible range of parameter values, given the data we've collected. On a Frequentist interpretation, I cannot assign any probability to an actual interval, as there is no more randomness left to apply a probability to.

So why do I care about this specific interval...what does it tell me about the parameter values being inferred? I've seen two main reactions to this question:

1. "Nothing!" it either does or does not cover the true parameter, but that's it.
2. "There is a 95% chance of the true value being in this interval"....usually followed by responses that "individual CI's cannot have a probability"

However, there is a third possibility that I think most of us who rely on confidence intervals actually utilize when interpreting an individual CI:

"I don't know if this particular 95% confidence interval contains the true value, but the procedure produces intervals that miss the mark only 5% of the time, so I will assume that this interval brackets the true value. Given this assumption, I still don't know which value in the interval is the true value, so I will interpret the interval as the "likely" values of the true value."

This (admittedly long winded...[not unlike this post ;-)]) interpretation has a number of nice features:

• It clearly separates the probability statement from the "subjective" or "likelihood" assessment. Therefore, I know that my interpretation will be wrong in 5% of samples (for a 95% CI)...but...it will be just fine in the other 95% (As Richard Royall memorably wrote "Sometimes the evidence is misleading."...following the data where it leads has its own risks!)
• There is a clear basis for evaluating/validating my confidence...just take a large number of samples from a known distribution and test if it works as advertised.

A Caveat...

• Some confidence intervals have a property called "ancillarity"...which means that the overall confidence in the procedure is really a marginal probability, where we are marginalizing over the conditional confidence of the procedure given the value of the ancillary statistic (hence, the ancillary statistic identifies "relevant subsets" of the population of possible confidence intervals, each of which may have a very different confidence compared to the overall average (i.e. unconditional) confidence of the procedure). There are methods to correct an interval given an ancillary statistic so that it achieves the desired confidence (look up "conditional inference" and the works of Nancy Reid, Richard Cox, and others)

Now, on to Bayesian Credibility Intervals

Bayesian estimates (probabilities, intervals) do not have to have any repeated-sampling properties. This is both a blessing and a curse. A blessing in that we can claim that the "probability" assigned to the interval is actually a probability for that interval. A curse in that we don't have any way to calibrate our sense of inferential risk - a Bayesian probability of "0.95" doesn't apply to anything tangible or verifiable. Yet, many statisticians rely on these intervals. So what am i missing? Here's my dilemma:

1. If we insist that "95%" probability doesn't apply to a theoretical sequence of repeated trials (or any sense of repetition), then "95%" is just a number calculated using a system that is formally consistent with the axioms of Probability Theory.
2. If we appeal to the track record of techniques using Bayesian Probability, then aren't we appealing to the frequentist criterion?

Now, I don't have any truck with Bayesian formulae for computing estimates...I see them as sensible attempts to "regularize" or "stabilize" small-sample estimates. However, I have yet to see a viable alternative to the concept of confidence when assessing inferential procedures. It just makes so much sense to me that we'd trust a method that is almost always correct.

Note that if we assume that the LLN holds and there is a true parameter value out there, then our Bayesian Credibility Interval is subject to the same tautology as a Confidence Interval...they either contain the true value or they do not. There is no logical way around that. It's just the "risk" we attach to the interval that seems to change.

I, for one, trust Confidence Intervals in much the same way I would trust a knowledgeable adviser. Most of the time, they are correct, but every so often they are wrong. Absent "ancillary" information (such as their accuracy conditional on the type of question being asked), I assume that their answers are correct and accept the (small) risk that I am making a mistake. That is the nature of randomness.

I don't know how I would have the same trust in an adviser who espouses a 95% Bayesian Probability of being right...

Answer 1

If you have accurately characterized your beliefs about a particular quantity in the prior distribution, then YES, you should have "confidence" in your updated beliefs, represented by the posterior distribution (and therefore credible intervals constructed from it), because Bayes' rule provides the appropriate way to update your beliefs upon seeing data.

The above statement is for one particular experiment, but more importantly, one particular quantity of interest. It says nothing about what happens to a whole set of experiments or quantities, nor should it, since this one particular quantity is the quantity of interest. It also says nothing about where the true parameter value is, but instead says something about where you believe the true parameter value is. So it says exactly what you should believe based on your choice of prior, your choice of likelihood, and the data you observed.

For coverage of confidence or credible intervals, we need to construct a repeatable statistical procedure. The above procedure is not repeatable because the prior construction is specific to the particular quantity of interest. But, we can construct default Bayesian procedures whose priors are constructed to satisfy certain properties. One of those properties is probability matching and credible intervals constructed bsaed on probability matching priors attain the appropriate frequentist coverage over repeated use of this prior.

So, if coverage gives you "confidence", then you should probably only use a default Bayesian procedure with a probability matching prior.

Answer 2

It sounds from your statement,"how can I have 95% belief in anything (you either believe something is true or you don't)," that you have a great deal of confidence in what statistics can tell us. Inherently, I (a proponent of Bayesian methods) ask how much to believe in something, and with new information, I am able to adjust my belief (adjust what I think). To me, belief simply represents how sure I am of the presence of an effect in combination with the totality of previous evidence, with "truth" never being given consideration.

For instance, I have an effect size, d, of 0.2 and a CI spanning from 0.004-0.510. How is it you can use this to derive a binary truth? What is your truth here? Is your sense of truth based on hypothetical resampling, from which a fixed but unknown parameter is captured by hypothetical intervals that are hypothetically constructed from the aforementioned resampling that has not actually occurred? In order for that to be the way in which truth is bestowed upon us, you would have to believe it to be so. I believe that to be improbable.

Based only on this effect size and interval, I would infer that, although the interval does not span zero, values very close to zero are probable. I would then pick a value based off past evidence to assess the posterior probability of an effect that can be considered important (practical significance). In this case, I would pick 0.2 (common in psychology), which would potentially result in a posterior probability of effect being less than 0.2 as 50 %, whereas there would also be a 50 % posterior probability of the effect being greater than 0.2. Based on this information, I would be very unsure of the importance of the effect, but would also consider the presence of an effect to be probable and worth consideration. For me, this is the appropriate inference based on the effect size and interval and can only be obtained using Bayesian methods.

As much as we would like a statistical programming language to give us all the answers, it cannot. We actually have to think and, from what we think, we form beliefs. As such, I would suggest giving serious consideration to the limits of statistical inference and to not be shy about being somewhat sure/unsure of what the truth actually is.