2
$\begingroup$

When trying to determine population variance from sample variance, it must be corrected by multiplying by: $$ \frac{n}{n-1} $$

and this is often explained intuitively as being because the mean of the sample is closer to the actual values of the sample than the mean of the population is.

From this explanation it would seem therefore that you could take two samples and use the mean of one to calculate the population variance from the other i.e.: $$ Sample 1: \big\{x_1, x_2, ..., x_i\big\} \\ Sample 2: \big\{y_1, y_2, ..., y_i\big\} \\ \sigma^2 = \frac{1}{n}\sum_1^n{(x_i - \bar{y})^2} $$

I can't think how you would prove/disprove this rigorously when you are considering different samples but testing it in python has shown it to give a variance greater than the actual population variance.

$\endgroup$
  • 1
    $\begingroup$ The correction gives an unbiased estimate of the population variance. There is no way to determine the population variance as an adjustment to the estimate. $\endgroup$ – Michael Chernick Nov 28 '17 at 23:38
  • 2
    $\begingroup$ It's hopeless to calculate the properties of one sample from those of another that is independent of it! What exactly are you trying to accomplish? Would you perhaps be trying to predict the variance of the second sample? $\endgroup$ – whuber Nov 28 '17 at 23:53
  • $\begingroup$ Suppose you got sample {x;y} and then randomly split it in halves {x}; {y}. Is there any sense to expect that the mean in y set is even closer to the population mean than the mean in x set? No sense. Because, by doing all that, we did not get the better than usual estimate of the population mean we cannot afford dropping that "-1" from the denominator. However, if y sample is much greater than x sample and is approaching the population in size, its mean is very close to the population mean; then yes, we may drop the "-1" in estimating variance from x values and that y mean. $\endgroup$ – ttnphns Nov 29 '17 at 5:38
  • $\begingroup$ I understand that the mean will be no closer to the actual population meam however the idea was that it shouldn't be as close to the values in the other sample as the mean of that sample. Also, as i said, the variance this way actually works out greater than the population variance contradicting that explanation $\endgroup$ – James Hogge Nov 29 '17 at 8:13
3
$\begingroup$

Let's assume the there are $n$ $x_i$ values and $m$ values in the sample used to calculate $\bar{y}$

What is the expected value of $\frac{1}{n}\sum_1^n (x_i - \bar{y})^2$? Expanding the squared term, we get:

$$(x_i - \bar{y})^2 = x_i^2 - 2x_i\bar{y} + \bar{y}^2$$

If we take the expected value of this (taking advantage of the linearity of expectation and the independence of $\bar{y}$ and each $x_i$) we get:

$$E[x_i^2] - 2E[x_i\bar{y}]+E[\bar{y}^2] = \sigma^2 + (E[x_i])^2 - 2E[x_i]E[\bar{y}] + E[\bar{y}^2] = $$ $$\sigma^2 + \mu^2 - 2\mu^2 + \frac{\sigma^2}{m} + \mu^2 = \sigma^2\left( 1+\frac{1}{m}\right) > \sigma^2\;\; \forall m<\infty$$

Therefore,

$$E\left[\frac{1}{n}\sum_1^n (x_i - \bar{y})^2\right] = \sigma^2\left( 1+\frac{1}{m}\right) = \sigma^2\left(\frac{m+1}{m}\right) $$

So, you have not removed the bias by using a second sample -- the bias comes in due to the variability of $\bar{x}$ or $\bar{y}$, not the fact they are "closer" to their sample values (a rather vague concept anyway). You'd need to multiply by $\left(\frac{m}{m+1}\right)$ to get back to an unbiased estimate.

$\endgroup$
  • $\begingroup$ Thanks! That makes perfect sense now. Not to mention matching the increase in variance I saw $\endgroup$ – James Hogge Nov 29 '17 at 8:17
0
$\begingroup$

The basic problem is that you are, as they say, 'using up one degree of freedom' when you use the sample mean to center the values in calculating the sample variance.

Imagine drawing three observations from a normal distribution with mean zero and standard deviation 1. Now, suppose those three observations are 0, 1, 1. Secretly, we all know that $\mu$ is 0, so the variance sum of the centered second-order terms is $1^2+1^2=2$.

But your estimated $\mu$ from the data is 0.667, or two-thirds. Thus, when you now calculate the sample estimator, the sum of those centered squares = $\frac{2}{3}^2+\frac{1}{3}^2+\frac{1}{3}^2 = \frac{2}{3}$.

In this case it is extreme, since we got two outliers fairly far from the true mean and one point right at it; so the estimate of the mean arithmetically makes the sum of sqaures of centered observations too small. That is why the $\frac{1}{n-1}$ shows up instead of just dividing by n itself.

Now, your idea seems sound on the surface. Why not use a separate sample from the same population to calculate the estimated mean, $\hat\mu$, so that you haven't used up a degree of freedom?

On the surface this sounds sensible, but two issues arise. First, in the real world we often only have n observations to work with - we can't get a whole new set of data. Second, when we can, we would prefer to pool it with the other data so that we have more accuracy in estimating all of the parameters. In other words, if we have two samples of 100, we will get more reliable results by treating it as one sample of 200 and using $\frac{1}{n-1}$ in calculating the estimated sample variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.