Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

In the wikipedia article on Credible Interval, it says:

For the case of a single parameter and data that can be summarised in a single sufficient statistic, it can be shown that the credible interval and the confidence interval will coincide if the unknown parameter is a location parameter (i.e. the forward probability function has the form Pr(x | μ) = f(x − μ) ), with a prior that is a uniform flat distribution;[5] and also if the unknown parameter is a scale parameter (i.e. the forward probability function has the form Pr(x | s) = f(x / s) ), with a Jeffreys' prior [5] — the latter following because taking the logarithm of such a scale parameter turns it into a location parameter with a uniform distribution. But these are distinctly special (albeit important) cases; in general no such equivalence can be made."

Could people give specific examples of this? When does the 95% CI actually correspond to "95% chance", thus "violating" the general definition of CI?

share|improve this question
add comment

1 Answer 1

up vote 6 down vote accepted

Take a normal distribution with known variance. We can take this variance to be 1 without losing generality (by simply dividing each observation by the square root of the variance). This has sampling distribution:

$$p(X_{1}...X_{N}|\mu)=\left(2\pi\right)^{-\frac{N}{2}}\exp\left(-\frac{1}{2}\sum_{i=1}^{N}(X_{i}-\mu)^{2}\right)=A\exp\left(-\frac{N}{2}(\overline{X}-\mu)^{2}\right)$$

Where $A$ is a constant which depends only on the data. This shows that the sample mean is a sufficient statistic for the population mean. If we use a uniform prior, then the posterior distribution for $\mu$ will be:

$$(\mu|X_{1}...X_{N})\sim Normal\left(\overline{X},\frac{1}{N}\right)\implies \left(\sqrt{N}(\mu-\overline{X})|X_{1}...X_{N}\right)\sim Normal(0,1)$$

So a $1-\alpha$ credible interval will be of the form:

$$\left(\overline{X}+\frac{1}{\sqrt{N}}L_{\alpha},\overline{X}+\frac{1}{\sqrt{N}}U_{\alpha}\right)$$

Where $L_{\alpha}$ and $U_{\alpha}$ are chosen such that a standard normal random variable $Z$ satisfies:

$$Pr\left(L_{\alpha}<Z<U_{\alpha}\right)=1-\alpha$$

Now we can start from this "pivotal quantity" for constructing a confidence interval. The sampling distribution of $\sqrt{N}(\mu-\overline{X})$ for fixed $\mu$ is a standard normal distribution, so we can substitute this into the above probability:

$$Pr\left(L_{\alpha}<\sqrt{N}(\mu-\overline{X})<U_{\alpha}\right)=1-\alpha$$

Then re-arrange to solve for $\mu$, and the confidence interval will be the same as the credible interval.

For scale parameters, the pdfs have the form $p(X_{i}|s)=\frac{1}{s}f\left(\frac{X_{i}}{s}\right)$. We can take the $(X_{i}|s)\sim Uniform(0,s)$, which corresponds to $f(t)=1$. The joint sampling distribution is:

$$p(X_{1}...X_{N}|s)=s^{-N}\;\;\;\;\;\;\;0<X_{1}...X_{N}<s$$

From which we find the sufficient statistic to be equal to $X_{max}$ (the maximum of the observations). We now find its sampling distribution:

$$Pr(X_{max}<y|s)=Pr(X_{1}<y,X_{2}<y...X_{N}<y|s)=\left(\frac{y}{s}\right)^{N}$$

Now we can make this independent of the parameter by taking $y=qs$. This means our "pivotal quantity" is given by $Q=s^{-1}X_{max}$ with $Pr(Q<q)=q^{N}$ which is the $beta(N,1)$ distribution. So, we can choose $L_{\alpha},U_{\alpha}$ using the beta quantiles such that:

$$Pr(L_{\alpha}<Q<U_{\alpha})=1-\alpha=U_{\alpha}^{N}-L_{\alpha}^{N}$$

And we substitute the pivotal quantity:

$$Pr(L_{\alpha}<s^{-1}X_{max}<U_{\alpha})=1-\alpha=Pr(X_{max}L_{\alpha}^{-1}>s>X_{max}U_{\alpha}^{-1})$$

And there is our confidence interval. For the Bayesian solution with jeffreys prior we have:

$$p(s|X_{1}...X_{N})=\frac{s^{-N-1}}{\int_{X_{max}}^{\infty}r^{-N-1}dr}=N (X_{max})^{N}s^{-N-1}$$ $$\implies Pr(s>t|X_{1}...X_{N})=N (X_{max})^{N}\int_{t}^{\infty}s^{-N-1}ds=\left(\frac{X_{max}}{t}\right)^{N}$$

We now plug in the confidence interval, and calculate its credibility

$$Pr(X_{max}L_{\alpha}^{-1}>s>X_{max}U_{\alpha}^{-1}|X_{1}...X_{N})=\left(\frac{X_{max}}{X_{max}U_{\alpha}^{-1}}\right)^{N}-\left(\frac{X_{max}}{X_{max}L_{\alpha}^{-1}}\right)^{N}$$

$$=U_{\alpha}^{N}-L_{\alpha}^{N}=Pr(L_{\alpha}<Q<U_{\alpha})$$

And presto, we have $1-\alpha$ credibility and coverage.

share|improve this answer
    
A masterpiece, thanks! I was hoping that there might be an answer like, "when calculating the mean of a sample from a Normal distribution, the 95% CI actually is also the 95% Credible Interval" or something simple like that. (Just making up this supposed answer, I have no clue as to specific examples.) –  Wayne Jul 2 '11 at 18:07
    
I believe that a frequentist 95% prediction/tolerance interval corresponds to a Bayesian prediction interval with OLS regression and normal errors. It appears so when I compare predict.lm's answer with a simulated answer, anyhow. Is that true? –  Wayne Jul 7 '11 at 18:16
    
For $Y=\alpha+\beta X$, If you use a uniform prior for $\alpha,\beta$ and jeffreys prior for $\sigma$, then you have equivalence. –  probabilityislogic Jul 8 '11 at 0:17
    
Great, thanks! I've been trying to explain a CI for a regression I've done in terms of a Confidence Interval, and it simply doesn't connect with a layman audience, which expects a Credible Interval. Makes life much easier for me... though perhaps it's bad for the overall statistical world, as it will reinforce the layman's misunderstanding of CI's. –  Wayne Jul 8 '11 at 14:21
    
@Wayne - the situation is a little more general than just location scale families. Usually a CI will be equivalent to credible interval, if it is based on a "sufficient statistic" (as these two were) where this exists. If no sufficient statistic, then CI needs to condition on what are called "ancillary statistics" to have credible interval interpretation. –  probabilityislogic Jul 9 '11 at 2:04
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.