# An unbiased estimate for population variance

Specifically, my notes claim that when calculating confidence intervals for population proportion $$p$$, the estimator for the population variance is given by $$\frac{P_sQ_s}{n}$$, where $$P_s=\frac{X}{n}$$ is the random variable for the proportion of successes. Thus we have $$E(P_s)=p$$ and Var$$(P_s)=\frac{pq}{n}$$. My question is, shouldn't we have $$\frac{P_sQ_s}{n} \times \frac{n}{n-1}$$ as the unbiased estimator instead?

Also, from my understanding, we say that $$T$$ is an (unbiased) estimator of the population parameter $$\theta$$ if we have $$E(T)=\theta$$. So is there any way to prove/disprove $$E(\frac{P_sQ_s}{n})=\sigma^2$$, where $$\sigma^2$$ is the population variance?

• Do your notes claim this is an unbiased estimator?
– whuber
May 2, 2020 at 18:13
• Nope it does not, I was just wondering if it was. Thanks so much for the answer! Just saw it now, will read through it carefully. May 3, 2020 at 5:34

Presumably $$Q_s(X) = 1 - P_s(X) = (n-X)/n.$$

Writing $$q=1-p$$, let's work out the expectation of $$n^2P_s(X)Q_s(X)$$ using the definition of expectation, the formula for Binomial probabilities, and the Binomial Theorem:

\eqalign{ E\left[n^2P_s(X)Q_s(X)\right] &= E\left[X(n-X)\right] \\ &= \sum_x \Pr(X=x)\, x(n-x) & \text{(Definition of expectation)} \\ &= \sum_{x=0}^n \binom{n}{x}p^x q^{n-x}\, x(n-x) &\text{(Binomial distribution)} \\ &=\sum_{x=0}^n \binom{n}{x}\, pq \frac{\partial^2}{\partial p\partial q} \left(p^x\,q^{n-x}\right) \\ &= pq \frac{\partial^2}{\partial p\partial q}\sum_{x=0}^n \binom{n}{x}\, p^x\,q^{n-x} & \text{(Linearity of differentiation)}\\ &= pq \frac{\partial^2}{\partial p\partial q}\left(p+q\right)^n &\text{(Binomial Theorem)}\\ &= pq\,n(n-1)(p+q)^{n-2}. }

(When $$n=1$$ or $$n=0$$ the result is just $$0.$$) Plugging in $$p+q=1$$ gives

$$E\left[n^2P_sQ_s\right] = n(n-1)pq$$

for all $$n,$$ whence for $$n\gt 1,$$

$$E\left[\frac{1}{n-1}\,P_s(X)Q_s(X)\right] = \frac{pq}{n}=\operatorname{Var}\left(P_s(X)\right).$$

Therefore $$P_s(X)Q_s(X)/(n-1)$$ is an unbiased estimator of the variance of $$X/n$$ (and so obviously $$P_s(X)Q_s(X)/n$$ is not: it is biased).

• Why then isn't the unbiased estimator used?
– Jon
Mar 17, 2021 at 17:19
• @Jon Since the parameter $p$ is estimated by the sample mean (in an unbiased way), and that in turn determines the population variance, it is rare for anyone to need a separate estimate of the population variance in a Binomial setting or to be able to do anything useful with it that cannot already be carried out with the estimate of $p.$
– whuber
Mar 17, 2021 at 17:58
• What about for example for a confidence interval for $p$?
– Jon
Mar 17, 2021 at 18:28
• @Jon Most of them just use the estimate of $p.$ (There are many confidence interval procedures for Binomial data: search our site for "Clopper" to find the best summaries.) Intuitively it wouldn't make much sense to employ a separate estimate of the variance that was inconsistent with the estimate of $p$--and the simple mathematical fact is that any estimate of the variance that is a (known) constant multiple of another will give the same procedure. The only thing that changes is how you describe it.
– whuber
Mar 17, 2021 at 18:34
• @darkgbm That's a standard technique in combinatorics. Research formal power series.
– whuber
Oct 4, 2023 at 13:12

Here's an approach using the following variance formula and rule

$$Var(\hat{p})=\frac{p(1-p)}{n}=E[\hat{p}^2]-E[\hat{p}]^2$$

where $$\hat{p}$$ is the sample proportion of times an indicator variable is 1 in a simple random sample of size $$n$$, i.e. the mean of an indicator variable, and $$p$$ is the corresponding population proportion for that indicator variable.

Suppose we estimate the population variance for that indicator variable, which is $$p(1-p)$$ (in terms of the population proportion $$p$$), using the estimator $$\hat{p}(1-\hat{p})$$ (which uses the sample statistics only). This estimator is biased and multiplying it by $$n/(n-1)$$ would make it unbiased.

Proof:

$$E[\hat{p}(1-\hat{p})]=E[\hat{p}-\hat{p}^2]=p-E[\hat{p}^2]$$

Rearranging the variance formula and rule above, we get $$E[\hat{p}^2]=p^2 + \frac{p(1-p)}{n}$$. Plugging that in we get

$$E[\hat{p}(1-\hat{p})]=p - p^2 - \frac{p(1-p)}{n} = \frac{n-1}{n}p(1-p)$$

• +1 Simple and elegant.
– whuber
Nov 17, 2021 at 18:13