Should I have "Confidence" in Credibility Intervals?

Preliminaries

First, I know that the Bayesian/Frequentist debate is rather long in tooth at this point, but I hope my question is sufficiently different from the others I reviewed on this site before I asked this question.

Lest others start forwarding me the numerous links on this site for "What is Bayesian Probability" or "Confidence Intervals vs Credibility Intervals", let me say that I am not concerned with the differences between Bayesian and Frequentist probabilities, as I understand that. What I want to know is why I should have any trust in Bayesian Credibility Intervals as a reliable description of the plausible range of parameter values.

My main question, then some background:

On what basis do we judge the reliability (or trustworthiness) of Bayesian Credibility Intervals?

I ask this because there is a conundrum with Credibility Intervals that I don't see with Confidence Intervals: how do I determine the 'risk' of being wrong?

An Example from Confidence Intervals

A 95% confidence interval is constructed to be right (i.e., cover the true parameter) in 95% of samples to which it is applied: the 95% is a probability that applies to the procedure over many samples. However, just like a realization of a random variable doesn't have to be anywhere near its mean value, any individual CI does not, in principle, have to bracket anything even close to the true value.

But...when we perform inference, we form a CI in an attempt to show the plausible range of parameter values, given the data we've collected. On a Frequentist interpretation, I cannot assign any probability to an actual interval, as there is no more randomness left to apply a probability to.

So why do I care about this specific interval...what does it tell me about the parameter values being inferred? I've seen two main reactions to this question:

1. "Nothing!" it either does or does not cover the true parameter, but that's it.
2. "There is a 95% chance of the true value being in this interval"....usually followed by responses that "individual CI's cannot have a probability"

However, there is a third possibility that I think most of us who rely on confidence intervals actually utilize when interpreting an individual CI:

"I don't know if this particular 95% confidence interval contains the true value, but the procedure produces intervals that miss the mark only 5% of the time, so I will assume that this interval brackets the true value. Given this assumption, I still don't know which value in the interval is the true value, so I will interpret the interval as the "likely" values of the true value."

This (admittedly long winded...[not unlike this post ;-)]) interpretation has a number of nice features:

• It clearly separates the probability statement from the "subjective" or "likelihood" assessment. Therefore, I know that my interpretation will be wrong in 5% of samples (for a 95% CI)...but...it will be just fine in the other 95% (As Richard Royall memorably wrote "Sometimes the evidence is misleading."...following the data where it leads has its own risks!)
• There is a clear basis for evaluating/validating my confidence...just take a large number of samples from a known distribution and test if it works as advertised.

A Caveat...

• Some confidence intervals have a property called "ancillarity"...which means that the overall confidence in the procedure is really a marginal probability, where we are marginalizing over the conditional confidence of the procedure given the value of the ancillary statistic (hence, the ancillary statistic identifies "relevant subsets" of the population of possible confidence intervals, each of which may have a very different confidence compared to the overall average (i.e. unconditional) confidence of the procedure). There are methods to correct an interval given an ancillary statistic so that it achieves the desired confidence (look up "conditional inference" and the works of Nancy Reid, Richard Cox, and others)

Now, on to Bayesian Credibility Intervals

Bayesian estimates (probabilities, intervals) do not have to have any repeated-sampling properties. This is both a blessing and a curse. A blessing in that we can claim that the "probability" assigned to the interval is actually a probability for that interval. A curse in that we don't have any way to calibrate our sense of inferential risk - a Bayesian probability of "0.95" doesn't apply to anything tangible or verifiable. Yet, many statisticians rely on these intervals. So what am i missing? Here's my dilemma:

1. If we insist that "95%" probability doesn't apply to a theoretical sequence of repeated trials (or any sense of repetition), then "95%" is just a number calculated using a system that is formally consistent with the axioms of Probability Theory.
2. If we appeal to the track record of techniques using Bayesian Probability, then aren't we appealing to the frequentist criterion?

Now, I don't have any truck with Bayesian formulae for computing estimates...I see them as sensible attempts to "regularize" or "stabilize" small-sample estimates. However, I have yet to see a viable alternative to the concept of confidence when assessing inferential procedures. It just makes so much sense to me that we'd trust a method that is almost always correct.

Note that if we assume that the LLN holds and there is a true parameter value out there, then our Bayesian Credibility Interval is subject to the same tautology as a Confidence Interval...they either contain the true value or they do not. There is no logical way around that. It's just the "risk" we attach to the interval that seems to change.

I, for one, trust Confidence Intervals in much the same way I would trust a knowledgeable adviser. Most of the time, they are correct, but every so often they are wrong. Absent "ancillary" information (such as their accuracy conditional on the type of question being asked), I assume that their answers are correct and accept the (small) risk that I am making a mistake. That is the nature of randomness.

I don't know how I would have the same trust in an adviser who espouses a 95% Bayesian Probability of being right...

• Perhaps your conundrum can be solved by noting that the Bayesian credible interval is not 'wrong' even if it does not cover the 'true' value of the parameter of interest. It doesn't claim to cover it, but instead it claims to contain (e.g.) 95% of the area under the posterior probability distribution. The accounting of 'error' that comes so easily with frequentist analyses is the most simple way to think about reliability and trustworthiness, but it's not the only way. Apr 4, 2016 at 1:03
• @MichaelLew thanks for your thoughtful comment. What I don't get is why I care about the posterior probability distribution? In fact, it doesn't just CLAIM to cover it, it actually does (by definition, correct?) Given a prior, a likelihood model, and the data, you will get a posterior that you can use to create intervals. But what is so great about this posterior...if I decide to act AS IF my parameter were in the credible interval, what how can I calibrate my risk of acting on an incorrect inference?
– user75138
Apr 4, 2016 at 14:54
• I note that the while the likelihood function is a product of the data and the statistical model, it is neither a model nor the model. Apr 5, 2016 at 21:57
• If your prior is based on an appropriate use of real relevant information then your Bayesian credible interval will often have better frequentist properties than the frequentist confidence interval. However, if your prior is uninformative (i.e. neutral regarding the location of the parameter of interest) the Bayesian interval will often perform about the same as the frequentist interval, and if your prior is bad then the Bayesian interval will behave worse. The Bayesian interval usually does not have a fixed frequentist calibration, but often performs well. Apr 5, 2016 at 22:01
• @MichaelLew thanks. I understood your second comment, but the first about likelihood not being a model or the model confused me.
– user75138
Apr 5, 2016 at 22:27

2 Answers

If you have accurately characterized your beliefs about a particular quantity in the prior distribution, then YES, you should have "confidence" in your updated beliefs, represented by the posterior distribution (and therefore credible intervals constructed from it), because Bayes' rule provides the appropriate way to update your beliefs upon seeing data.

The above statement is for one particular experiment, but more importantly, one particular quantity of interest. It says nothing about what happens to a whole set of experiments or quantities, nor should it, since this one particular quantity is the quantity of interest. It also says nothing about where the true parameter value is, but instead says something about where you believe the true parameter value is. So it says exactly what you should believe based on your choice of prior, your choice of likelihood, and the data you observed.

For coverage of confidence or credible intervals, we need to construct a repeatable statistical procedure. The above procedure is not repeatable because the prior construction is specific to the particular quantity of interest. But, we can construct default Bayesian procedures whose priors are constructed to satisfy certain properties. One of those properties is probability matching and credible intervals constructed bsaed on probability matching priors attain the appropriate frequentist coverage over repeated use of this prior.

So, if coverage gives you "confidence", then you should probably only use a default Bayesian procedure with a probability matching prior.

• (+1) for the reference to probability matching and your generally helpful post! So, what does the 5% NOT included in a 95% Cred. Interval represent? In fact, how can I have 95% belief in anything (you either believe something is true or you don't). Now, I get the idea of a Likelihood interval as the interval of "likely values" (subject to some cutoff), but the idea that I can dole out belief and then act on that as if I've learned something about objective world is unsettling to me...
– user75138
Apr 4, 2016 at 15:04
• As you can see..I think that inference essentially boils down to "feelings" about risk. The main question is how we quantify that risk and use it in "statistical decisionmaking" via a loss-functions
– user75138
Apr 4, 2016 at 15:20
• @Bey "how can I have 95% belief in anything (you either believe something is true or you don't)" -- this is a puzzling assertion. If you ask me what I believe is the population of Finland, I can roughly sketch a probability distribution describing my beliefs. I will not be very certain, so the PDF will be pretty broad. Then I could draw an interval that encloses 95% of the area around the mode. I will have 95% belief that Finland's population lies in this range. Apr 17, 2016 at 22:37
• Yes, you seem to be confused about what Bayesian statistic is trying to do. There is no concept of "risk" there, it's a frequentist issue. If you want to have an explicit control of your error rate, then you should use frequentist statistics that is designed for that purpose. Apr 18, 2016 at 12:27
• @amoeba (+1) thank you. That is the clearest answer I've received so far.
– user75138
Apr 18, 2016 at 14:09

It sounds from your statement,"how can I have 95% belief in anything (you either believe something is true or you don't)," that you have a great deal of confidence in what statistics can tell us. Inherently, I (a proponent of Bayesian methods) ask how much to believe in something, and with new information, I am able to adjust my belief (adjust what I think). To me, belief simply represents how sure I am of the presence of an effect in combination with the totality of previous evidence, with "truth" never being given consideration.

For instance, I have an effect size, d, of 0.2 and a CI spanning from 0.004-0.510. How is it you can use this to derive a binary truth? What is your truth here? Is your sense of truth based on hypothetical resampling, from which a fixed but unknown parameter is captured by hypothetical intervals that are hypothetically constructed from the aforementioned resampling that has not actually occurred? In order for that to be the way in which truth is bestowed upon us, you would have to believe it to be so. I believe that to be improbable.

Based only on this effect size and interval, I would infer that, although the interval does not span zero, values very close to zero are probable. I would then pick a value based off past evidence to assess the posterior probability of an effect that can be considered important (practical significance). In this case, I would pick 0.2 (common in psychology), which would potentially result in a posterior probability of effect being less than 0.2 as 50 %, whereas there would also be a 50 % posterior probability of the effect being greater than 0.2. Based on this information, I would be very unsure of the importance of the effect, but would also consider the presence of an effect to be probable and worth consideration. For me, this is the appropriate inference based on the effect size and interval and can only be obtained using Bayesian methods.

As much as we would like a statistical programming language to give us all the answers, it cannot. We actually have to think and, from what we think, we form beliefs. As such, I would suggest giving serious consideration to the limits of statistical inference and to not be shy about being somewhat sure/unsure of what the truth actually is.

• Thanks for the input! I think you read a bit too much into my comment to one of the posters. I don' t think I have any more or less confidence (belief?) in what statistics can tell us. This question is about how we come to trust Credibility Intervals absent a way to calibrate them. Also, there are three things to consider with inference: what do the data say, what should we believe, what should we do? My concern is the relationship between the last two.
– user75138
Apr 17, 2016 at 17:28
• Here's a very simple, albeit somewhat cliche, example: You are lost, and you ask someone for help on which direction to go. Based on their demeanor, body language, etc, you feel 95% sure that they are telling the truth. Now, they either are or are not, so you will need to decide (or just sit there and starve to death). If you take their advice, then you believe they are, in fact, telling the truth (you have "closed" the last 5% of "sureness"). Now, why did being 95% sure make you feel justified in following their advice...what makes you think your sureness tracks truth?
– user75138
Apr 17, 2016 at 17:32
• I rereading this post, I completely agree that I responded to your response to another comment. I will leave my comment (I will edit it some), but will respond to the original post as well. In response to another comment, I often make decisions based on being somewhat sure I am making the right decision. We can only do the best we can based on the data and past evidence, which probably always falls short of being 100 % sure. Apr 17, 2016 at 17:33
• On a related note...what is the difference between "sureness" and belief? What does it mean to be 95% "sure"?
– user75138
Apr 17, 2016 at 17:34
• I do not know. I actually do not think there is ever a 100 % correct answer or a model that is exactly correct. I do the best I can, which is usually all we can do. Apr 17, 2016 at 17:44