Why does this not give population variance?

Question

When trying to determine population variance from sample variance, it must be corrected by multiplying by: $$ \frac{n}{n-1} $$

and this is often explained intuitively as being because the mean of the sample is closer to the actual values of the sample than the mean of the population is.

From this explanation it would seem therefore that you could take two samples and use the mean of one to calculate the population variance from the other i.e.: $$ Sample 1: \big\{x_1, x_2, ..., x_i\big\} \\ Sample 2: \big\{y_1, y_2, ..., y_i\big\} \\ \sigma^2 = \frac{1}{n}\sum_1^n{(x_i - \bar{y})^2} $$

I can't think how you would prove/disprove this rigorously when you are considering different samples but testing it in python has shown it to give a variance greater than the actual population variance.

Answer 1

Let's assume the there are $n$ $x_i$ values and $m$ values in the sample used to calculate $\bar{y}$

What is the expected value of $\frac{1}{n}\sum_1^n (x_i - \bar{y})^2$? Expanding the squared term, we get:

$$(x_i - \bar{y})^2 = x_i^2 - 2x_i\bar{y} + \bar{y}^2$$

If we take the expected value of this (taking advantage of the linearity of expectation and the independence of $\bar{y}$ and each $x_i$) we get:

$$E[x_i^2] - 2E[x_i\bar{y}]+E[\bar{y}^2] = \sigma^2 + (E[x_i])^2 - 2E[x_i]E[\bar{y}] + E[\bar{y}^2] = $$ $$\sigma^2 + \mu^2 - 2\mu^2 + \frac{\sigma^2}{m} + \mu^2 = \sigma^2\left( 1+\frac{1}{m}\right) > \sigma^2\;\; \forall m<\infty$$

Therefore,

$$E\left[\frac{1}{n}\sum_1^n (x_i - \bar{y})^2\right] = \sigma^2\left( 1+\frac{1}{m}\right) = \sigma^2\left(\frac{m+1}{m}\right) $$

So, you have not removed the bias by using a second sample -- the bias comes in due to the variability of $\bar{x}$ or $\bar{y}$, not the fact they are "closer" to their sample values (a rather vague concept anyway). You'd need to multiply by $\left(\frac{m}{m+1}\right)$ to get back to an unbiased estimate.

Answer 2

The basic problem is that you are, as they say, 'using up one degree of freedom' when you use the sample mean to center the values in calculating the sample variance.

Imagine drawing three observations from a normal distribution with mean zero and standard deviation 1. Now, suppose those three observations are 0, 1, 1. Secretly, we all know that $\mu$ is 0, so the variance sum of the centered second-order terms is $1^2+1^2=2$.

But your estimated $\mu$ from the data is 0.667, or two-thirds. Thus, when you now calculate the sample estimator, the sum of those centered squares = $\frac{2}{3}^2+\frac{1}{3}^2+\frac{1}{3}^2 = \frac{2}{3}$.

In this case it is extreme, since we got two outliers fairly far from the true mean and one point right at it; so the estimate of the mean arithmetically makes the sum of sqaures of centered observations too small. That is why the $\frac{1}{n-1}$ shows up instead of just dividing by n itself.

Now, your idea seems sound on the surface. Why not use a separate sample from the same population to calculate the estimated mean, $\hat\mu$, so that you haven't used up a degree of freedom?

On the surface this sounds sensible, but two issues arise. First, in the real world we often only have n observations to work with - we can't get a whole new set of data. Second, when we can, we would prefer to pool it with the other data so that we have more accuracy in estimating all of the parameters. In other words, if we have two samples of 100, we will get more reliable results by treating it as one sample of 200 and using $\frac{1}{n-1}$ in calculating the estimated sample variance.