First, I know that the Bayesian/Frequentist debate is rather long in tooth at this point, but I hope my question is sufficiently different from the others I reviewed on this site before I asked this question.
Lest others start forwarding me the numerous links on this site for "What is Bayesian Probability" or "Confidence Intervals vs Credibility Intervals", let me say that I am not concerned with the differences between Bayesian and Frequentist probabilities, as I understand that. What I want to know is why I should have any trust in Bayesian Credibility Intervals as a reliable description of the plausible range of parameter values.
My main question, then some background:
On what basis do we judge the reliability (or trustworthiness) of Bayesian Credibility Intervals?
I ask this because there is a conundrum with Credibility Intervals that I don't see with Confidence Intervals: how do I determine the 'risk' of being wrong?
An Example from Confidence Intervals
A 95% confidence interval is constructed to be right (i.e., cover the true parameter) in 95% of samples to which it is applied: the 95% is a probability that applies to the procedure over many samples. However, just like a realization of a random variable doesn't have to be anywhere near its mean value, any individual CI does not, in principle, have to bracket anything even close to the true value.
But...when we perform inference, we form a CI in an attempt to show the plausible range of parameter values, given the data we've collected. On a Frequentist interpretation, I cannot assign any probability to an actual interval, as there is no more randomness left to apply a probability to.
So why do I care about this specific interval...what does it tell me about the parameter values being inferred? I've seen two main reactions to this question:
- "Nothing!" it either does or does not cover the true parameter, but that's it.
- "There is a 95% chance of the true value being in this interval"....usually followed by responses that "individual CI's cannot have a probability"
However, there is a third possibility that I think most of us who rely on confidence intervals actually utilize when interpreting an individual CI:
"I don't know if this particular 95% confidence interval contains the true value, but the procedure produces intervals that miss the mark only 5% of the time, so I will assume that this interval brackets the true value. Given this assumption, I still don't know which value in the interval is the true value, so I will interpret the interval as the "likely" values of the true value."
This (admittedly long winded...[not unlike this post ;-)]) interpretation has a number of nice features:
- It clearly separates the probability statement from the "subjective" or "likelihood" assessment. Therefore, I know that my interpretation will be wrong in 5% of samples (for a 95% CI)...but...it will be just fine in the other 95% (As Richard Royall memorably wrote "Sometimes the evidence is misleading."...following the data where it leads has its own risks!)
- There is a clear basis for evaluating/validating my confidence...just take a large number of samples from a known distribution and test if it works as advertised.
- Some confidence intervals have a property called "ancillarity"...which means that the overall confidence in the procedure is really a marginal probability, where we are marginalizing over the conditional confidence of the procedure given the value of the ancillary statistic (hence, the ancillary statistic identifies "relevant subsets" of the population of possible confidence intervals, each of which may have a very different confidence compared to the overall average (i.e. unconditional) confidence of the procedure). There are methods to correct an interval given an ancillary statistic so that it achieves the desired confidence (look up "conditional inference" and the works of Nancy Reid, Richard Cox, and others)
Now, on to Bayesian Credibility Intervals
Bayesian estimates (probabilities, intervals) do not have to have any repeated-sampling properties. This is both a blessing and a curse. A blessing in that we can claim that the "probability" assigned to the interval is actually a probability for that interval. A curse in that we don't have any way to calibrate our sense of inferential risk - a Bayesian probability of "0.95" doesn't apply to anything tangible or verifiable. Yet, many statisticians rely on these intervals. So what am i missing? Here's my dilemma:
- If we insist that "95%" probability doesn't apply to a theoretical sequence of repeated trials (or any sense of repetition), then "95%" is just a number calculated using a system that is formally consistent with the axioms of Probability Theory.
- If we appeal to the track record of techniques using Bayesian Probability, then aren't we appealing to the frequentist criterion?
Now, I don't have any truck with Bayesian formulae for computing estimates...I see them as sensible attempts to "regularize" or "stabilize" small-sample estimates. However, I have yet to see a viable alternative to the concept of confidence when assessing inferential procedures. It just makes so much sense to me that we'd trust a method that is almost always correct.
Note that if we assume that the LLN holds and there is a true parameter value out there, then our Bayesian Credibility Interval is subject to the same tautology as a Confidence Interval...they either contain the true value or they do not. There is no logical way around that. It's just the "risk" we attach to the interval that seems to change.
I, for one, trust Confidence Intervals in much the same way I would trust a knowledgeable adviser. Most of the time, they are correct, but every so often they are wrong. Absent "ancillary" information (such as their accuracy conditional on the type of question being asked), I assume that their answers are correct and accept the (small) risk that I am making a mistake. That is the nature of randomness.
I don't know how I would have the same trust in an adviser who espouses a 95% Bayesian Probability of being right...