10
$\begingroup$

I was reading an entry level statistics textbook. In the chapter on maximum likelihood estimation of the success proportion in data with binomial distribution, it gave a formula for calculating a confidence interval and then nonchalantly mentioned

Consider its actual coverage probability, that is, the probability that the method produces an interval that captures the true parameter value. This may be quite a bit less than the nominal value.

And goes on with a suggestion to construct an alternative "confidence interval", which presumably contains the actual coverage probability.

I was confronted with the idea of nominal and actual coverage probability for the first time. Making my way through old questions here, I think I got an understanding for it: there are two different concepts we call probability, the first being how probable it is that a not-yet-happened event will produce a given result, and the second is how probable is that an observing agent's guess for the result of an already-happened-event is true. It also seemed that confidence intervals only measure the first type of probability, and that something called "credible intervals" measure the second type of probability. I summarily assumed that confidence intervals are the ones which calculate "nominal coverage probability" and credible intervals are the ones which cover "actual coverage probability".

But maybe I have misinterpreted the book (it is not entirely clear on whether the different calculation methods it offers are for a confidence interval and a credible interval, or for two different types of confidence interval), or the other sources I used to come to my current understanding. Especially a comment which I got on another question,

Confidence intervals for frequentist, credible for Bayesian

made me doubt my conclusions, as the book did not describe a Bayesian method in that chapter.

So please clarify if my understanding is correct, or if I have made a logical error on the way.

Improve this question
$\endgroup$
1
  • $\begingroup$ The nominal coverage probability is the "target" coverage probability: the one we try to attain when we derive a method providing a confidence interval. The actual coverage is the "true" coverage. Some people say that the confidence interval is exact when the actual coverage equals the nominal coverage. Scotchi and Unwisdom have mentionned that the confidence interval is never exact for discrete data. Another example is when we use an asymptotic confidence interval: it is exact only when $n \to \infty$. I totally understand your idea because "actual" is also a synonym of "present". $\endgroup$
    – Stéphane Laurent
    Commented Feb 12, 2014 at 21:29

3 Answers 3

Reset to default
6
$\begingroup$

In general, the actual coverage probability will never be equal to the nominal probability when you are working with a discrete distribution.

The confidence interval is defined as a function of the data. If you are working with the binomial distribution, there are only finitely many possible outcomes ($ n+1$ to be precise), so there are only finitely many possible confidence intervals. Since the parameter $ p $ is continuous, it's pretty easy to see that the coverage probability (which is a function of $ p $) can do no better than be approximately 95% (or whatever).

It is generally true that methods based on the CLT will have coverage probabilities below the nominal value, but other methods can actually be more conservative.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Here's a useful formal statement of the definition: Given a sample space $\langle\Omega,\mathcal{F},P\rangle$ and an unknown parameter $\theta$, a $1-\alpha$ confidence procedure consists of a pair of functions $L\leq U:\Omega\to\mathbb{R}$ such that $$P\big[\left\{\omega\in\Omega\vert [L(\omega),U(\omega)]\ni\theta\right\}\big]\approx 1-\alpha.$$ The left hand side of this expression is the $\textit{coverage probability}$ (note that this depends on θ) and the RHS is the nominal confidence level. If the infimum (over $\Omega$) of the LHS is equal to the RHS then the procedure is exact. $\endgroup$
    – Unwisdom
    Commented Feb 7, 2014 at 18:23
8
$\begingroup$

It's nothing to do with Bayesian credible intervals vs frequentist confidence intervals. A 95% (say) confidence interval is defined as giving at least 95% coverage whatever the true value of the parameter $\pi$. So when the nominal coverage is 95%, the actual coverage may be 97% when $\pi=\pi_1$, 96.5% when $\pi=\pi_2$, but for no value of $\pi$ is it less than 95%. The issue (i.e. a discrepancy between nominal & actual coverage) arises with discrete distributions like the binomial.

As an illustration, consider observing $x$ successes from $n$ binomial trials with unknown success probability $\pi$: $$ \begin{array}{c,c,c} x & \pi_\mathrm{U} & \Pr(X= x | \pi=0.7) & I(\pi_\mathrm{U}\leq 0.7)\\ 0 & 0.3930378 & 0.000729 & 0\\ 1 & 0.5818034 & 0.010206 & 0\\ 2 & 0.7286616 & 0.059535 & 1\\ 3 & 0.8468389 & 0.185220 & 1\\ 4 & 0.9371501 & 0.324135 & 1\\ 5 & 0.9914876 & 0.302526 & 1\\ 6 & 1.0000000 & 0.117649 & 1\\ \end{array} $$ The first column shows the possible observed values of $x$. The second shows the exact† $95\%$ upper‡ confidence bound $\pi_\mathrm{U} =\pi: [\Pr(X>x | \pi)=0.95]$ that you would calculate in each case. Now suppose $\pi=0.7$: the third column shows the probability of each observed value of $x$ under this supposition; the fourth shows for which cases the calculated confidence interval covers the true parameter value, flagging them with a $1$. If you add up the probabilities for the cases in which the confidence interval does cover the true value you get the actual coverage, $0.989065$. For different true values of $\pi$, the actual coverage will be different:

coverages

The nominal coverage is only achieved when the true parameter values coincide with the obtainable upper bounds.

[I just re-read your question & noticed that the author says the actual may be less than the nominal coverage probability. So I reckon they're talking about an approximate method for calculating the confidence interval, though what I said above still goes. The graph might suggest reporting an average confidence level of about $98\%$ but—averaging over values of an unknown parameter?]

† Exact in the sense that the actual coverage is never less than the nominal coverage for any value of $\pi$, & equal to it for some values of $\pi$— @Unwisdom's sense, not @Stephane's.

‡ Intervals with upper & lower bounds are more commonly used of course; but a little more complicated to explain, & there's only one exact interval to consider with just an upper bound. (See Blaker (2000), "Confidence curves and improved exact confidence intervals for discrete distributions", Canadian Journal of Statistics, 28, 4 & the references.)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for answering. Now that I know what the actual coverage probability is, do you have a guess why the user in this question was sent to questions which explain the difference between credible and confidence intervals? This is where I got the idea that the actual/nominal coverage prob. duality is related. stats.stackexchange.com/questions/63922/… $\endgroup$
    – rumtscho
    Commented Feb 11, 2014 at 0:27
  • $\begingroup$ Probably because the OP only gives a link to where he's seen the terms "nominal" & "actual" (rather than summarizing or quoting from it in the question as you did), & then devotes the rest of his question to his misinterpretation of their use in that context. $\endgroup$
    – Scortchi ♦
    Commented Feb 11, 2014 at 9:38
3
$\begingroup$

I think the difference is actually about the use of approximations made when calculating confidence intervals. For example if we use the fairly standard CI of

$$\text{estimate}\pm 1.96 \times \text {estimated standard error}$$

We may call this a "95% confidence interval". However, is it usually the case that several approximations are made here. If we don't make the approximations, then we can calculate the actual coverage. A typical situation is under estimating the standard error. Then the intervals are too narrow to capture the true value with 95% probability. They might only capture the true value with say 85% probability. The "actual coverage" probability might be calculated using a monte carlo simulation of some kind (eg generate $1000$ sample data sets using a chosen true value, then calculate 95% CI for each, and find that $850$ actually contained the true value).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.