9
$\begingroup$

The standard error of a proportion will be the largest it can be for a given N when the proportion in question is 0.5, and gets smaller the further the proportion is from 0.5. I can see why this is so when I look at the equation for the standard error of a proportion, but I can't explain this any further.

Is there an explanation beyond the mathematical properties of the formula? If so, why is there less uncertainty around estimated proportions (for a given N) as they get closer to 0 or 1?

$\endgroup$
7
$\begingroup$

Background and Terminology

To be perfectly clear what we're discussing, let's establish some concepts and terminology. A nice model for proportions is the binary urn: it contains balls colored either silver ("success") or fuchsia ("failure"). The proportion of silver balls in the urn is $p$ (but this is not the "proportion" we will be talking about).

This urn provides a way to model a Bernoulli Trial. To obtain one realization, thoroughly mix the balls and blindly draw one out, observing its color. To obtain additional realizations, first reconstitute the box by returning the drawn ball, then repeat the procedure a predetermined number of times. The sequence of $n$ realizations can be summarized by the count of its successes, $X$. It is a random variable whose properties are completely determined by $n$ and $p$. The distribution of $X$ is called a Binomial$(n,p)$ distribution. The (experimental, or "sample") proportion is the ratio $X/n$.

Figure

These figures are barplots of probability distributions for various binomial proportions $X/n$. Most noteworthy is a consistent pattern, regardless of $n$, in which the distributions become narrower (and the bars correspondingly higher) as $p$ moves from $1/2$ on down.

The standard deviation of $X/n$ is the standard error of proportion mentioned in the question. For any given $n$, this quantity can depend only on $p$. Let's call it $\operatorname{se}(p)$. By switching the roles of the balls--call the silver ones "failures" and the fuchsia ones "successes"--it's easy to see that $\operatorname{se}(p) = \operatorname{se}(1-p)$. Thus the situation where $p=1-p$--that is, $p=1/2$--must be special. The question concerns how $\operatorname{se}(p)$ varies as $p$ moves away from $1/2$ towards a more extreme value, such as $0$.

Knowledge vs Understanding

Because everyone has been shown figures like these early in their education, everybody "knows" the widths of the plots--which are measured by $\operatorname{se}(p)$--must decrease as $p$ moves away from $1/2$. But that knowledge is really just experience, whereas the question seeks a deeper understanding. Such understanding is available from a careful analysis of Binomial distributions, such as Abraham de Moivre undertook some 300 years ago. (They were akin in spirit to those I presented in a discussion of the Central Limit Theorem.) I think, though, that some relatively simple considerations might suffice to make the point that the widths must be widest near $p=1/2$.

A Simple Intuitive Analysis

It is clear that we should expect the proportion of successes in the experiment to be close to $p$. The standard error concerns how far from that expectation we might reasonably suppose the actual outcome $X/n$ will lie. Supposing, without any loss of generality, that $p$ is between $0$ and $1/2$, what would it take to increase $X/n$ from $p$? Typically, around $pn$ of the balls drawn in an experiment were silver and (therefore) around $(1-p)n$ were fuchsia. To get more silver balls, some of those $p n$ fuchsia outcomes had to have differed. How likely is it that chance could operate in this way? The obvious answer is that when $p$ is small, it's never very likely that we're going to draw a silver ball. Thus, our chances of drawing silver balls instead of fuchsia ones are always low. We might reasonably hope that by pure luck, a proportion $p$ of the fuchsia outcomes could have differed, but it seems unlikely that many more than that would have changed. Thus, it is plausible that $X$ would not vary by much more than $p\times (1-p)n$. Equivalently, $X/n$ would not vary by much more than $p(1-p)n/n = p(1-p)$.

The Denouement

Thus the magic combination $p(1-p)$ appears. This virtually settles the question: obviously this quantity peaks at $p=1/2$ and decreases to zero at $p=0$ or $p=1$. It provides an intuitive yet quantitative justification for assertions that "one extreme is more limiting than the other" or other such efforts to describe what we know.

However, $p(1-p)$ is not quite the correct value: it merely points the way, telling us what quantity ought to matter for estimating the spread of $X$. We have ignored the fact that luck also tends to act against us: just as some of the fuchsia balls could have been silver, some of the silver balls could have been fuchsia. Accounting for all the possibilities rigorously can get complicated, but the upshot is that instead of using $p(1-p)n$ as a reasonable limit on how much $X$ could deviate from its expectation $pn$, to account for all the possible outcomes properly we have to take the square root $\sqrt{p(1-p)n}$. (For a more careful account of why, please visit (https://stats.stackexchange.com/a/3904.) Dividing by $n$, we learn that random variations of the proportion $X/n$ itself should be on the order of $\sqrt{p(1-p)n}/n = \sqrt{\frac{p(1-p)}{n}},$ which is the standard error of $X/n$.

$\endgroup$
3
$\begingroup$

Consider the function p(1-p) for 0<=p<=1. Using calculus you can see that at p=1/2 it is 1/4 which is the maximum value. If you can see that this is for the binomial related to the standard deviation of the estimate of the proportion which is sqrt(p(1-p)/n) then p=1/2 is the maximum. When p=1 or 0 the standard error is 0 because you will always get all 1s or all 0s respectively. So as you get close to 0 or 1 a continuity argument says that the standard error approaches 0 as p approaches 0 or 1. In fact it decreases monotonically as p approaches 0 or 1. For large n the estimated proportion should be close to the actual proportion.

$\endgroup$
  • 3
    $\begingroup$ The OP has already remarked that "I can see why this is so when I look at the equation for the standard error of a proportion." Therefore I believe they are asking not for an analysis of the formula $p(1-p)$, but rather for a deeper understanding of why the formula--whatever it might be--really ought to be maximized at $p=1/2$. $\endgroup$ – whuber Jun 8 '17 at 15:08
  • 1
    $\begingroup$ @whuber I answered the way I did because I see that the formula is fundamental to the understanding of why the variance is largest at p=1/2 and very small when p is near 0 or 1. Maybe it is best to say that there is no explanation completely devoid from the formula. $\endgroup$ – Michael Chernick Jun 8 '17 at 16:59
1
$\begingroup$

The binomial distribution tends to be roughly symmetric (for large $n$ it is approximately normal).

Since the ratio must be between 0 and 1, the uncertainty will be constrained by these bounds. Unless the mean ratio is exactly in the middle, one of these bounds will be more limiting than the other.

For a symmetric unimodal bell curve centered at $p$ to fit into the unit interval, its half-width must be less than $\min[\,p\,,1-p\,]$.

$\endgroup$
  • $\begingroup$ Yes--but the other bound will be less limiting! Why don't the two effects cancel? $\endgroup$ – whuber Jun 8 '17 at 15:08
  • $\begingroup$ @whuber I was arguing from symmetry (i.e. in the simple "large $n$" case, the symmetric bell curve must fit into the interval, so its half-width is constrained by the tighter side, $\min[p,1-p]$) $\endgroup$ – GeoMatt22 Jun 8 '17 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.