Return to Answer

I think it has been pretty much covered by whuber, but I just wish to expand on the use of $n-1$; where it comes from and whether it applies here.

In an ordinary sample variance, many people use an $n-1$ denominator to make the usual sum-of-squares-based variance estimate unbiased (not everyone prefers unbiasedness to other properties though). This is called Bessel's correction but appears to have been derived by Gauss. A simple derivation is here

Presumably whoever wrote that formula has concluded that the same should be done with the usual variance estimate for a binomial proportion, which is generally estimated as $p(1-p)/n$ (where $p$ is the sample proportion).

Can we see whether the expectation of the usual estimator of variance is the population value?

Take $\pi$ to be the corresponding population proportion. That is, does $\text{E}[p(1-p)/n]=\pi(1-\pi)/n$?

Equivalently, does $\text{E}[p(1-p)]=\pi(1-\pi)$?

Note that if $X$ is the observed count, $p = X/n$, where under the usual sampling assumptions, $X\sim \text{binomial}(n,\pi)$.

\begin{eqnarray} \text{E}[p(1-p)] &=& 1/n^2 {E}(X(n-X))\\ &=& 1/n^2 (nEX - EX^2) \\ &=& 1/n^2 (n^2\pi - n\pi(1-\pi) - n^2\pi^2 )\\ &=& 1/n^2 (n^2\pi - n\pi +n\pi^2 - n^2\pi^2 )\\ &=& 1/n^2 \cdot n\pi(n - 1 +\pi - n\pi )\\ &=& 1/n^2 \cdot n\pi(n - 1)(1-\pi)\\ &=& \frac{n-1}{n} \pi (1-\pi) \end{eqnarray}

Hence $\text{E}[p(1-p)/(n-1)]=\pi (1-\pi)/n$

It looks like (assuming I made no errors) it is the case here too - that the usual estimator of the variance of the proportion is biased, and may be unbiased by multiplying the typical estimator by $\frac{n}{n-1}$.

(Edit: In retrospect this is obvious; one simply need apply the ordinary bias calculation for a sample variance to a sample of 0's and 1's)

Which means it appears that the formula you have has been chosen to give an unbiased estimate.

(I wonder why people seem happy to use a biased variance estimate for binomials when there's such an insistence on using an unbiased one in other situations. I have no good answer for that; I'll continue using biased estimators whenever it makes sense to me.)

I think it has been pretty much covered by whuber, but I just wish to expand on the use of $n-1$; where it comes from and whether it applies here.

In an ordinary sample variance, many people use an $n-1$ denominator to make the usual sum-of-squares-based variance estimate unbiased (not everyone prefers unbiasedness to other properties though). This is called Bessel's correction but appears to have been derived by Gauss. A simple derivation is here

Presumably whoever wrote that formula has concluded that the same should be done with the usual variance estimate for a binomial proportion, which is generally estimated as $p(1-p)/n$ (where $p$ is the sample proportion).

Can we see whether the expectation of the usual estimator of variance is the population value?

Take $\pi$ to be the corresponding population proportion. That is, does $\text{E}[p(1-p)/n]=\pi(1-\pi)/n$?

Equivalently, does $\text{E}[p(1-p)]=\pi(1-\pi)$?

Note that if $X$ is the observed count, $p = X/n$, where under the usual sampling assumptions, $X\sim \text{binomial}(n,\pi)$.

\begin{eqnarray} \text{E}[p(1-p)] &=& 1/n^2 {E}(X(n-X))\\ &=& 1/n^2 (nEX - EX^2) \\ &=& 1/n^2 (n^2\pi - n\pi(1-\pi) - n^2\pi^2 )\\ &=& 1/n^2 (n^2\pi - n\pi +n\pi^2 - n^2\pi^2 )\\ &=& 1/n^2 \cdot n\pi(n - 1 +\pi - n\pi )\\ &=& 1/n^2 \cdot n\pi(n - 1)(1-\pi)\\ &=& \frac{n-1}{n} \pi (1-\pi) \end{eqnarray}

Hence $\text{E}[p(1-p)/(n-1)]=\pi (1-\pi)/n$

It looks like (assuming I made no errors) it is the case here too - that the usual estimator of the variance of the proportion is biased, and may be unbiased by multiplying the typical estimator by $\frac{n}{n-1}$.

(Edit: In retrospect this is obvious; one simply need apply the ordinary bias calculation for a sample variance to a sample of 0's and 1's)

Which means it appears that the formula you have has been chosen to give an unbiased estimate.

(I wonder why people seem happy to use a biased variance estimate for binomials when there's such an insistence on using an unbiased one in other situations. I have no good answer for that; I'll continue using biased estimators whenever it makes sense to me, which seems to be rather more often than most people do.)

I think it has been pretty much covered by whuber, but I just wish to expand on the use of $n-1$; where it comes from and whether it applies here.

In an ordinary sample variance, many people use an $n-1$ denominator to make the usual sum-of-squares-based variance estimate unbiased (not everyone prefers unbiasedness to other properties though). This is called Bessel's correction but appears to have been derived by Gauss. A simple derivation is here

Presumably whoever wrote that formula has concluded that the same should be done with the usual variance estimate for a binomial proportion, which is generally estimated as $p(1-p)/n$ (where $p$ is the sample proportion).

Can we see whether the expectation of the usual estimator of variance is the population value?

Take $\pi$ to be the corresponding population proportion. That is, does $\text{E}[p(1-p)/n]=\pi(1-\pi)/n$?

Equivalently, does $\text{E}[p(1-p)]=\pi(1-\pi)$?

Note that if $X$ is the observed count, $p = X/n$, where under the usual sampling assumptions, $X\sim \text{binomial}(n,\pi)$.

\begin{eqnarray} \text{E}[p(1-p)] &=& 1/n^2 {E}(X(n-X))\\ &=& 1/n^2 (nEX - EX^2) \\ &=& 1/n^2 (n^2\pi - n\pi(1-\pi) - n^2\pi^2 )\\ &=& 1/n^2 (n^2\pi - n\pi +n\pi^2 - n^2\pi^2 )\\ &=& 1/n^2 \cdot n\pi(n - 1 +\pi - n\pi )\\ &=& 1/n^2 \cdot n\pi(n - 1)(1-\pi)\\ &=& \frac{n-1}{n} \pi (1-\pi) \end{eqnarray}

Hence $\text{E}[p(1-p)/(n-1)]=\pi (1-\pi)/n$

It looks like (assuming I made no errors) it is the case here too - that the usual estimator of the variance of the proportion is biased, and may be unbiased by multiplying the typical estimator by $\frac{n}{n-1}$.

Which means it appears that the formula you have has been chosen to give an unbiased estimate.

(I wonder why people seem happy to use a biased variance estimate for binomials when there's such an insistence on using an unbiased one in other situations. I have no good answer for that; I'll continue using biased estimators whenever it makes sense to me.)

I think it has been pretty much covered by whuber, but I just wish to expand on the use of $n-1$; where it comes from and whether it applies here.

In an ordinary sample variance, many people use an $n-1$ denominator to make the usual sum-of-squares-based variance estimate unbiased (not everyone prefers unbiasedness to other properties though). This is called Bessel's correction but appears to have been derived by Gauss. A simple derivation is here

Presumably whoever wrote that formula has concluded that the same should be done with the usual variance estimate for a binomial proportion, which is generally estimated as $p(1-p)/n$ (where $p$ is the sample proportion).

Can we see whether the expectation of the usual estimator of variance is the population value?

Take $\pi$ to be the corresponding population proportion. That is, does $\text{E}[p(1-p)/n]=\pi(1-\pi)/n$?

Equivalently, does $\text{E}[p(1-p)]=\pi(1-\pi)$?

Note that if $X$ is the observed count, $p = X/n$, where under the usual sampling assumptions, $X\sim \text{binomial}(n,\pi)$.

\begin{eqnarray} \text{E}[p(1-p)] &=& 1/n^2 {E}(X(n-X))\\ &=& 1/n^2 (nEX - EX^2) \\ &=& 1/n^2 (n^2\pi - n\pi(1-\pi) - n^2\pi^2 )\\ &=& 1/n^2 (n^2\pi - n\pi +n\pi^2 - n^2\pi^2 )\\ &=& 1/n^2 \cdot n\pi(n - 1 +\pi - n\pi )\\ &=& 1/n^2 \cdot n\pi(n - 1)(1-\pi)\\ &=& \frac{n-1}{n} \pi (1-\pi) \end{eqnarray}

Hence $\text{E}[p(1-p)/(n-1)]=\pi (1-\pi)/n$

It looks like (assuming I made no errors) it is the case here too - that the usual estimator of the variance of the proportion is biased, and may be unbiased by multiplying the typical estimator by $\frac{n}{n-1}$.

(Edit: In retrospect this is obvious; one simply need apply the ordinary bias calculation for a sample variance to a sample of 0's and 1's)

Which means it appears that the formula you have has been chosen to give an unbiased estimate.

(I wonder why people seem happy to use a biased variance estimate for binomials when there's such an insistence on using an unbiased one in other situations. I have no good answer for that; I'll continue using biased estimators whenever it makes sense to me.)

I think it has been pretty much covered by whuber, but I just wish to expand on the use of $n-1$; where it comes from and whether it applies here.

In an ordinary sample variance, many people use an $n-1$ denominator to make the usual sum-of-squares-based variance estimate unbiased (not everyone prefers unbiasedness to other properties though). This is called Bessel's correction but appears to have been derived by Gauss. A simple derivation is here

Presumably whoever wrote that formula has concluded that the same should be done with the usual variance estimate for a binomial proportion, which is generally estimated as $p(1-p)/n$ (where $p$ is the sample proportion).

Can we see whether the expectation of the usual estimator of variance is the population value?

Take $\pi$ to be the corresponding population proportion. That is, does $\text{E}[p(1-p)/n]=\pi(1-\pi)/n$?

Equivalently, does $\text{E}[p(1-p)]=\pi(1-\pi)$?

Note that if $X$ is the observed count, $p = X/n$, where under the usual sampling assumptions, $X\sim \text{binomial}(n,\pi)$.

\begin{eqnarray} \text{E}[p(1-p)] &=& 1/n^2 {E}(X(n-X))\\ &=& 1/n^2 (nEX - EX^2) \\ &=& 1/n^2 (n^2\pi - n\pi(1-\pi) - n^2\pi^2 )\\ &=& 1/n^2 (n^2\pi - n\pi +n\pi^2 - n^2\pi^2 )\\ &=& 1/n^2 \cdot n\pi(n - 1 +\pi - n\pi )\\ &=& 1/n^2 \cdot n\pi(n - 1)(1-\pi)\\ &=& \frac{n-1}{n} \pi (1-\pi) \end{eqnarray}

Hence $\text{E}[p(1-p)/(n-1)]=\pi (1-\pi)/n$

It looks like (assuming I made no errors) it is the case here too - that the usual estimator of the variance of the proportion is biased, and may be unbiased by multiplying the typical estimator by $\frac{n}{n-1}$.

Which means it appears that the formula you have has been chosen to give an unbiased estimate.

(I wonder why people seem happy to use a biased variance estimate for binomials when there's such an insistence on using an unbiased one in other situations. I have no good answer for that; I'll continue using biased estimators whenever it makes sense to me.)

I think it has been pretty much covered by whuber, but I just wish to expand on the use of $n-1$; where it comes from and whether it applies here.

In an ordinary sample variance, many people use an $n-1$ denominator to make the usual sum-of-squares-based variance estimate unbiased (not everyone prefers unbiasedness to other properties though). This is called Bessel's correction but appears to have been derived by Gauss. A simple derivation is here

Presumably whoever wrote that formula has concluded that the same should be done with the usual variance estimate for a binomial proportion, which is generally estimated as $p(1-p)/n$ (where $p$ is the sample proportion).

Can we see whether the expectation of the usual estimator of variance is the population value?

Take $\pi$ to be the corresponding population proportion. That is, does $\text{E}[p(1-p)/n]=\pi(1-\pi)/n$?

Equivalently, does $\text{E}[p(1-p)]=\pi(1-\pi)$?

Note that if $X$ is the observed count, $p = X/n$, where under the usual sampling assumptions, $X\sim \text{binomial}(n,\pi)$.

\begin{eqnarray} \text{E}[p(1-p)] &=& 1/n^2 {E}(X(n-X))\\ &=& 1/n^2 (nEX - EX^2) \\ &=& 1/n^2 (n^2\pi - n\pi(1-\pi) - n^2\pi^2 )\\ &=& 1/n^2 (n^2\pi - n\pi +n\pi^2 - n^2\pi^2 )\\ &=& 1/n^2 \cdot n\pi(n - 1 +\pi - n\pi )\\ &=& 1/n^2 \cdot n\pi(n - 1)(1-\pi)\\ &=& \frac{n-1}{n} \pi (1-\pi) \end{eqnarray}

Hence $\text{E}[p(1-p)/(n-1)]=\pi (1-\pi)/n$

It looks like (assuming I made no errors) it is the case here too - that the usual estimator of the variance of the proportion is biased, and may be unbiased by multiplying the typical estimator by $\frac{n}{n-1}$.

Which means it appears that the formula you have has been chosen to give an unbiased estimate.

(I wonder why people seem happy to use a biased variance estimate for binomials when there's such an insistence on using an unbiased one in other situations. I have no good answer for that; I'll continue using biased estimators whenever it makes sense to me.)