# Frequentist confidence intervals = constant trapping probability?

In the case of estimating an unknown mean of a normal distribution with known variance, if I'm not mistaken, the confidence interval contains $\theta$ with probability $1 - \alpha$, regardless of the actual value of $\theta$. In other cases (e.g. when the variance is not necessarily constant), is it still the case that the confidence interval contains the actual value with probability $1 - \alpha$? Or is it only the case that for any value of $\theta$, the confidence interval contains the actual value with probability at least $1 - \alpha$? (which if I'm not mistaken is how confidence intervals are defined).

Thanks!

• When you say 'constant variance', do you mean 'known variance'? For non-constant variance, unknown variance? Nov 20, 2012 at 11:30
• Ah yes, I indeed meant known variance. I've edited the question, thanks! Nov 21, 2012 at 15:44

It depends on how strictly you want to define it, most I believe would accept that for all parameter values, if the interval has at least 1 - alpha coverage - it can be taken as a confidence interval (no matter how you came up with it). There is an interesting post on this right now at Normal Deviate

One could define it more strictly as always equal to 1 - alpha (similar I think I recall the technical term being) which is sometime achieved by post study randomization (or something equivalent) or not admitting relevant subsets or etc.

Not a bad idea to do a simulation and plot the coverge over the parameter space. For instance, the odds ratio from a two group experiment with binary outcomes has a very interesting plot.

As per comment below

choose your CI to make the coverage 1−α for some parameters and greater for others?

• this is usual case. Normaldeviate's plot was an example with just one parameter and that is very misleading for when there are more than one parameter. But there are always caveats - for instance the whole real line, plane, etc has 100% covergae for all parameter values. One of the more famous unresolved problems is very simple problem to state and dicussed on wiki. Behrens-Fisher.

Doing some simulations and plotting coverage over the full parameter space is highly recomended ;-)

• Thanks! This question was actually prompted by that post, I saw the horizontal red line representing CI coverage and wondered if this line is always horizontal, or if the (normal + known variance) case is a special case. I guess what I was interested in is not so much about the definition of CI, but more like, in general, are there cases where it is impossible for the coverage to be $1-\alpha$ for all parameter values, and so you have to choose your CI to make the coverage $1 - \alpha$ for some parameters and greater for others? (I'm guessing yes, but confirmation would be nice). Nov 21, 2012 at 15:56
• en.wikipedia.org/wiki/Behrens%E2%80%93Fisher_problem Nov 21, 2012 at 16:18

The classic definition of a confidence interval $I_\theta$ for the unknown parameter $\theta$ is indeed that $\mbox{P}(\theta\in I_\theta)\geq 1-\alpha$ for all $\theta$. This definition is needed for theoretical comparisons of confidence intervals (i.e. when deriving optimality results), but is used less frequently in applied statistics, where confidence intervals commonly only satisify $\mbox{P}(\theta\in I_\theta)\approx 1-\alpha$, with $\mbox{P}(\theta\in I_\theta)< 1-\alpha$ for some $\theta$.

Examples of intervals of the latter type are bootstrap confidence intervals, which are very useful when the sampling distribution of $X_1,\ldots,X_n$ is either unknown or too complicated to be treated analytically.

Another example of where $\mbox{P}(\theta\in I_\theta)\neq 1-\alpha$ is confidence intervals for the parameter $p$ in the binomial distribution $\mbox{Bin}(n,p)$. A classical confidence interval is the Clopper-Pearson interval, which is such that $\mbox{P}(p\in I_p)\geq 1-\alpha$ for all $p\in(0,1)$. Below is the coverage $\mbox{P}(p\in I_p)$ as a function of $p$ when $n=25$ and $\alpha=0.05$:

For symmetry ("equivariance") reasons, the coverage for $1-p$ is the same as that for $p$.

An alternative, quite popular, interval is the Wilson score interval, which only satisfies $\mbox{P}(p\in I_p)\approx 1-\alpha$ but is prefered since it has significantly shorter length than the Clopper-Pearson interval (most practitioners are willing to accept slight undercoverage for some values of $p$ if they can obtain shorter intervals):

As you can see, the answer to your question depends on whether you use the term confidence interval in the classic sense or in the sense that it is used in modern applied statistics.

(Confidence intervals for $p$ with coverage $1-\alpha$ for all values of $p$ can be obtained by using randomization. Essentially, this means that the function $\mbox{P}(p\in I_p)$ is smoothed by adding additional randomness to the analysis. This is however not all that popular in practice.)