10
$\begingroup$

I'm trying to understand the following text currently (i.e., 2019-09-25) in Wikipedia about the Clopper-Pearson interval:

The Clopper–Pearson interval is an early and very common method for calculating binomial confidence intervals.[8] This is often called an 'exact' method, because it is based on the cumulative probabilities of the binomial distribution (i.e., exactly the correct distribution rather than an approximation). However, in cases where we know the population size, the intervals may not be the smallest possible, because they include impossible proportions: for instance, for a population of size 10, an interval of [0.35, 0.65] would be too large as the true proportion cannot lie between 0.35 and 0.4, or between 0.6 and 0.65.

I do understand that in the given example it would be impossible to get an outcome that would represent a binomial proportion of 0.35 (as this would require 3.5 successes, which is not a possible outcome).

However, I believe the CP-interval is meant to represent the range of underlying probabilities of success (the 'true proportions') that have some minimum probability to produce the observed (integer) outcome. As far as I can see, these 'true proportions' can take values between 0.35 and 0.4, or between 0.6 and 0.65.

Am I seeing this wrong, or is the cited text incorrect?


UPDATE

Upon reflection I can see where my confusion originates from. In the context I'm working in, we have $N$ clients that have a $p$ probability that something is happening to them. Our observable is the number of clients that have actually encountered this event.

So, we are not sampling from a vase with green and red balls, and the values that $p$ can take on are fully independent of $N$ (they are definitely not integer multiples of $1/N$). I guess, the cited Wikipedia text refers to the vase case, although the introduction of the page seems to be very much applicable to my situation as well:

In statistics, a binomial proportion confidence interval is a confidence interval for the probability of success calculated from the outcome of a series of success–failure experiments (Bernoulli trials). In other words, a binomial proportion confidence interval is an interval estimate of a success probability p when only the number of experiments n and the number of successes nS are known.

Whereas I assumed Wikipedia was referring to my client group of size $N$ as the population, they were actually referring to the size of the vase.

In that light I can now understand the restrictions on the confidence interval and the discussion on the efficiency of the CP interval. Given the continuous nature of my probability value I don't think any of the restrictions of the CP method apply to my case.

$\endgroup$
  • 1
    $\begingroup$ This quotation is bizarre. It must contemplate sampling small populations of size $n$ with replacement, effectively contemplating a model in which the binomial parameter $p$ is constrained to a discrete subset of $[0,1];$ namely, $\{0,1/n,\ldots,k/n,\ldots, 1\}.$ But that's a different model and there's little point either to introducing it or to criticizing any binomial procedure based on it. Usually "finite-population" problems are those involving sampling without replacement, but then they involve hypergeometric probabilities, not binomial ones. $\endgroup$ – whuber Sep 25 '19 at 20:33
  • $\begingroup$ @whuber, this is also stated on wikipedia.... "For instance, it can also be applied to the case where the samples are drawn without replacement from a population of a known size, instead of repeated draws of a binomial distribution. In this case, the underlying distribution would be the hypergeometric distribution. " $\endgroup$ – Sextus Empiricus Sep 30 '19 at 11:27
  • $\begingroup$ Gents, I made an essential update to my question. Seems like my operational context lead me astray a little bit. Thanks for your contributions so far. I really appreciated it. $\endgroup$ – Sjoerd C. de Vries Sep 30 '19 at 17:57
  • $\begingroup$ The Wikipedia article is (at that point) indeed speaking about population size instead of sample size. It is easy to confuse. I did this myself too. Still, the population size interpretation remains weird. It is not common (especially the specific example of sampling a population of size ten), and at the same time, the common issue with the PC interval is the discreteness and the idea that the interval true coverage rate might be higher. The "problem" with the limited population size is also not really a big deal (a bit trivial) and also occurs with the other frequentist intervals. $\endgroup$ – Sextus Empiricus Sep 30 '19 at 18:10
  • $\begingroup$ "Given the continuous nature of my probability value I don't think any of the restrictions of the CP method apply to my case." You will still have that the true coverage rate of the PC interval may be higher. This issue with the coverage is different from the issue with known population size. $\endgroup$ – Sextus Empiricus Sep 30 '19 at 18:17
3
+50
$\begingroup$

There are multiple concepts blended together into that statement on Wikipedia.

  • Let's first tackle the statement "the intervals may not be the smallest possible"
  • Then look at the different issue "cases where we know the population size"

Point 1

the intervals may not be the smallest possible

It is correct that the intervals are not efficient (too conservative). But for different reasons.


The Clopper-Pearson interval contains those values of $p$ for which an hypothesis test at a certain $\alpha$ level would pass (not reject the null). (this is how confidence intervals work)

The problem with the binomial distribution is in this last part "an hypothesis test at a certain $\alpha$". As the outcome variable is discrete and so will be the p-values associated with those outcomes.

Say you wish to test for a coin flipping proces the hypothesis p=0.5 (with a confidence interval you do this for all values of p) with a level $\alpha=0.01$ and you flip eight times then the observations, if the hypothesis would be true would be distributed as

0     1     2     3     4     5     6     7     8
0.004 0.032 0.109 0.219 0.273 0.219 0.109 0.032 0.004

The values 1 untill 7 would make you not reject the hypothesis at this preset 1% level. But this acceptance (ie. the not rejection) would occur 99.2% of the time and not 99% (if p=0.5 would be true).

So this means that the confidence interval will be conservative. It will be containing the actual parameter value more often than stated. The Clopper-Pearson interval will guarantee that the interval contains the parameter at least $x$ percent of the time, but it will not need to be exactly that percentage.

This relates to the further text on wikipedia

For example, the true coverage rate of a 95% Clopper–Pearson interval may be well above 95%, depending on n and θ. Thus the interval may be wider than it needs to be to achieve 95% confidence

Point 2

cases where we know the population size

This refers to sampling from a population with a (known) fixed size. E.g. sampling with replacement from a vase with $n$ balls, out of which a proportion $p$ has a certain characteristic (say red). Obviously, this proportion needs to be a multiple of $1/n$. There can only be an integer number of red balls in the vase.

Say you have a vase with ten balls and wish to predict how many are red by sampling it. Then it makes no sense to speak of the proportion of number of red balls is between 0.35 and 0.65 (equivalent to 'the number of red balls in the vase is between 3.5 and 6.5').


So there are two issues that make the Clopper-Pearson interval not the smallest possible. This may have been inconveniently mixed together on Wikipedia or not spelled out very clearly.

The first point actually only tells that the interval is not efficient; it may have a higher true coverage rate. But there is no solution to make the interval smaller. Except when we use prior knowledge and can use Bayesian intervals.

The second point is a bit weird. It is not a typical situation to sample small populations with replacement. I believe that it may have been used as an argument for the inefficiency of the intervals, while instead the first point was meant/supposed to be used.


Written by StackExchangeStrike

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Isn't CP suboptimal because it calculates c.i.s symmetric in the two tails? $\endgroup$ – Michael M Sep 30 '19 at 11:34
  • $\begingroup$ @MichaelM I do not believe that is the main issue. If you like you could compute one-side CP intervals. But maybe I did not get your comment. How do you actually consider CP intervals as symmetric? $\endgroup$ – Sextus Empiricus Sep 30 '19 at 11:38
  • $\begingroup$ "Symmetric" in the sense of "central", i.e. assigning $\alpha/2$ error probability to each tail. "Central" is the better word, sorry. $\endgroup$ – Michael M Sep 30 '19 at 11:52
  • 1
    $\begingroup$ @MartijnWeterings See the update on my question above. Due to a perhaps mutual misunderstanding we were talking past each other. Apologies for that. $\endgroup$ – Sjoerd C. de Vries Sep 30 '19 at 18:05
  • 1
    $\begingroup$ @MartijnWeterings: Thanks for the explanation. Very helpful! $\endgroup$ – Michael M Sep 30 '19 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.