3
$\begingroup$

I am examining differences between groups with two categorical variables. For one category of categorical variable #1 all values are in one category of variable #2. Thus the proportion of category in variable #1 that is in the selected category of variable #2 is 100%. Is it theoretically possible to calculate a confidence interval around that proportion?

I case it matters, I am currently using R and the "Survey" and "Srvyr" packages, with the survey_mean() function and I get NaN values for those CIs.

EDIT: I should add that the sryvr package has the function survey_mean() and in that function i've specified that I want "beta" methods for calculating the intervals. survey_mean() passes to svyciprop() from Thomas Lumley's Survey package. It says in the help files "All methods undercover for probabilities close enough to zero or one, but "beta", "likelihood", "logit", and "logit" are noticeably better than the other two. None of the methods will work when the observed proportion is exactly 0 or 1."

So that explains why I received the NaN.

$\endgroup$
1

1 Answer 1

6
$\begingroup$

There are many kinds of confidence intervals for binomial proportions. The Wikipedia article on this topic, discusses some of them.

Some of the kinds of confidence intervals do not work well with data that show sample proportions near $0$ or $1.$ Especially if you have such sample results, you need to avoid types of intervals that give error messages, non-answers, or absurd answers.

Wald intervals. These are 'asymptotic' intervals based on assumptions that are strictly true only as $n$ approaches $\infty$ and so they do not work well for small $n.$ In particular if the sample proportion is near $0$ or $1$ they may have "nonsense" boundaries that lie outside $[0,1].$

For example: if we have $x = 39$ successes in $n = 40$ trials, the point estimate of $p$ is $\hat p = x/n = 39/40 = 0.975$ and the 95% Wald interval is $\hat p \pm 1.96 \sqrt{\frac{\hat p(1-\hat p)}{n}}.$ which computes to $(0.927, 1.023).$ [Computation below in R.] Also, in case $x = n = 40$ you can check that the Wald interval is of zero length, as if to 'guarantee' that $p=1,$ which is inappropriate.

p.est = 39/40
CI = p.est + qnorm(c(.025,.975))*sqrt(p.est*(1-p.est)/40); CI
[1] 0.9266173 1.0233827

Jeffries intervals. These are based on a Bayesian argument that begins with the non-informative prior distribution $\mathsf{Beta}(.5, .5).$ This guarantees that the result, treated as a frequentist confidence interval, can never have endpoints outside the unit interval. [Even though this interval estimate has a Bayesian 'heritage', the Wikipedia article says that it has excellent properties when used as a frequentist CI; this claim matches my own personal experience.]

A 95% Jeffries CI for data $x = n = 40$ uses quantiles $.025$ and $.975$ of the distribution $\mathsf{Beta}(.5+x,\, .5+n-x),$ so that the interval computes to $(0.9395, 0.99999).$

qbeta(c(.025,.975), 40.5, .5) 
[1] 0.9395020 0.9999878

Clopper-Pearson intervals. The R procedure binom.test gives an 'exact' confidence interval. [The interval is called exact because it relies on binomial CDFs, avoiding normal and other approximations. Its somewhat messy formula is shown in the Wikipedia link.]

For $x = 39, n = 40$ the resulting 95% CI amounts to $(0.8684, 0.9994),$ as shown below. For $x = n = 40,$ the 95% CI is $(0.9119, 1.0000),$ essentially a one-sided CI [computation not shown].

binom.test(39,40)$conf.int
[1] 0.8684141 0.9993673
attr(,"conf.level")
[1] 0.95

Note: For about the last 25 years an Agresti-Coull modification of the Wald interval has been recommended. In order to emulate other more accurate kinds of CIs, it artificially appends 2 successes and two failures to the data and then uses the formula for the Wald interval.

In many cases this interval does give more accurate results than the Wald interval. However, for sample proportions at or near $0$ or $1,$ Agresti-Coull intervals can still produce bounds outside of the unit interval.

I mention this style of interval because it is frequently used and I suspect it may be giving the results you show in your answer. Without an actual numerical example, one can only speculate.

$\endgroup$
5
  • $\begingroup$ @thomaslumley. Any ideas on this? $\endgroup$
    – BruceET
    Commented Apr 21, 2021 at 5:41
  • $\begingroup$ Very nice and comprehensive answer, +1! I wonder whether it would make sense to migrate it to the highly-voted thread Confidence interval for Bernoulli sampling, so we can close this one as a duplicate of that one and have everything in one place. What do you think? Do you want to contact the mods about this? $\endgroup$ Commented Apr 21, 2021 at 6:41
  • $\begingroup$ @StephanKolassa. Migration is OK with me. This question includes asking specifically about NaNs from an R procedure Thomas Lumley wrote, so might wait a couple of days to see if he wants to give a direct answer to that issue here. $\endgroup$
    – BruceET
    Commented Apr 21, 2021 at 14:55
  • 1
    $\begingroup$ The NaNs very probably come from the fact that there is perfect separation/fitted probabilities 0 and 1, so your answer is spot on. $\endgroup$ Commented Apr 21, 2021 at 15:04
  • 1
    $\begingroup$ Thanks, for this nice description. I've edited my post to provide additional context. I'm already using "beta" methods for calculating the intervals, but still get NaN. $\endgroup$
    – RNB
    Commented Apr 21, 2021 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.