2
$\begingroup$

In a population, there is a proportion $p$ that I want to know. I sample the population and observe proportion $\hat{p}$. It is very unlikely that $p=\hat{p}$, and what I want to do is say something in terms of probability about $p$ being close to $\hat{p}$. I want a method of creating what is called a "95% confidence interval" around $\hat{p}$ that satisfies the following:

If this is method repeated indefinitely, with probability 1 the proportion of intervals produced that contain $p$ converges to 0.95

Everything I've ever seen tells me that the right way to construct this interval is according to the following formula, $$ \hat{p} \pm 1.96\sqrt\frac{\hat{p}(1-\hat{p})}{n} \qquad (A) $$

My question regards the validity of the $\sqrt\frac{\hat{p}(1-\hat{p})}{n}$ part of this formula. It seems to me that constructing the interval this way does not satisfy the condition written in bold above.

Argument

Suppose $p=0.5$ and $n$ is large enough that we can assume normality without going far wrong. In this case it is true that $\hat{p}$ has a 95% chance of being within the following interval, $$ 0.5 \pm 1.96\sqrt{\frac{0.5(1-0.5)}{n}} \quad = \quad 0.5 \pm \frac{1.96(0.5)}{n} $$
Let me call this interval $(0.5-Q,0.5+Q)$. The maximum value of $x(1-x)$ occurs at $x=0.5$. This implies that when $\hat{p} = 0.5 + Q$ the interval constructed according to (A) does not contain $p$; the radius of the interval is smaller than Q. The size of the interval about $\hat{p}$ constructed according to (A) varies continuously with $\hat{p}$, so this implies that there is some $\epsilon>0$ such that no $\hat{p}$ in the interval $(0.5 + Q-\epsilon, 0.5 +Q)$ leads to an interval containing $p$, constructed according to (A). Ipso facto, the probability that the interval constructed according to (A) contains $p$ is less than 95%.

In the following picture, there is a 95% chance that $\hat{p}$ will be between the brackets, but those $\hat{p}$ lying in the blue region lead to intervals that do not contain $p$.

enter image description here

When $p$ is larger than 0.5 we can still consider the interval $(p-Q, p+Q)$ in which $\hat{p}$ falls with 95% probability. It is possible that $p-Q$ is further away from $0.5$ than $p$, in which case the situation is essentially the same as when $p=0.5$ and it is not true that the interval produced around $\hat{p}$ will contain $p$ with 95% probability. It is also possible that $p-Q$ is closer to 0.5 than $p$. In this case, utilizing (A) leads to $\hat{p}$ slightly less than $p+Q$ whose intervals don't contain $p$ (pictured in blue below), as well as $\hat{p}$ slightly less than $p-Q$ whose intervals DO contain $p$ (pictured in red below)

enter image description here

In this case the red region is larger than the blue region. I am not able to say for sure, but if was a betting man I would wager that what is gained in the red does not exactly balance what is lost in the blue; i.e. the probability that the interval produced according to (A) contains $p$ is still not 95%.

Alternative

As an alternative, it seems the more responsible thing to do would be to use 0.5 always in the computation of the confidence interval. i.e. $$ \hat{p} \pm 1.96\sqrt{\frac{0.25}{n}} $$ This is gives the biggest possible interval. The above bold condition is still not satisfied, but the following condition is,

If this method is repeated indefinitely, with probability 1 the proportion of intervals produced that contain $p$ converges to at least 0.95

Why is this alternative approach never used? I'm willing to accept that the answer to my question is something along the lines of "It's close enough, especially when $n$ is large. Formula (A) is not strictly correct, but nobody cares. Everything is just approximations anyway." I just want to make sure there is not something I'm missing or some error in my thinking. I've only every seen (A) presented as if it has rigorous mathematical backing.

$\endgroup$
1
  • 1
    $\begingroup$ This is a well-studied problem, and the text-book interval is known to have poor properties. Unfortunately, I have not found a comprehensive introduction how to derive confidence intervals in almost all practical cases (proportions, mean values, arbitrary estimators), so I had to write my own survey, which you might find helpful (section 3 should answer your question): lionel.kr.hsnr.de/~dalitz/data/publications/… $\endgroup$
    – cdalitz
    Jun 15, 2020 at 10:34

1 Answer 1

3
$\begingroup$

Well, "It's close enough, especially when $n$ is large"

In fact, people often do use better approximations when $n$ isn't very large. Some of them are described here on Wikipedia. The Clopper-Pearson interval (described there) always has at least its claimed coverage probability, and is a lot shorter than using p=0.5.

There's one setting where using $p=0.5$ actually is popular: election polling. The "maximum margin of error" that pollers quote is 1.96 times the standard error of $\hat p$ when $p=0.5$.

Election polling also shows the problem with overly-conservative intervals: if you are in a country where minor parties are important, you might want a confidence interval for a party with, say, 5% support. In a sample of 1000 people, the maximum margin of error is about 3%, so the interval would be 2%-8%, which is too wide. Worse, for a party with 2% support the interval would be -1% to 5%.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.