21
$\begingroup$
  • What does it mean to say that "the variance is a biased estimator".
  • What does it mean to convert a biased estimate to an unbiased estimate through a simple formula. What does this conversion do exactly?
  • Also, What is the practical use of this conversion? Do you convert these scores when using certain kind of statistics?
$\endgroup$
22
$\begingroup$

You can find everything here. However, here is a brief answer.

Let $\mu$ and $\sigma^2$ be the mean and the variance of interest; you wish to estimate $\sigma^2$ based on a sample of size $n$.

Now, let us say you use the following estimator:

$S^2 = \frac{1}{n} \sum_{i=1}^n (X_{i} - \bar{X})^2$,

where $\bar{X} = \frac{1}{n} \sum_{i=1}^n X_i$ is the estimator of $\mu$.

It is not too difficult (see footnote) to see that $E[S^2] = \frac{n-1}{n}\sigma^2$.

Since $E[S^2] \neq \sigma^2$, the estimator $S^2$ is said to be biased.

But, observe that $E[\frac{n}{n-1} S^2] = \sigma^2$. Therefore $\tilde{S}^2 = \frac{n}{n-1} S^2$ is an unbiased estimator of $\sigma^2$.

Footnote

Start by writing $(X_i - \bar{X})^2 = ((X_i - \mu) + (\mu - \bar{X}))^2$ and then expand the product...

Edit to account for your comments

The expected value of $S^2$ does not give $\sigma^2$ (and hence $S^2$ is biased) but it turns out you can transform $S^2$ into $\tilde{S}^2$ so that the expectation does give $\sigma^2$.

In practice, one often prefers to work with $\tilde{S}^2$ instead of $S^2$. But, if $n$ is large enough, this is not a big issue since $\frac{n}{n-1} \approx 1$.

Remark Note that unbiasedness is a property of an estimator, not of an expectation as you wrote.

$\endgroup$
  • 1
    $\begingroup$ I mean more in theoretical terms. I can find the formula in any book, but i'm interested more in the explanation in words. The expectation of the sigma is unbiased and we can transform the estimate into the expectation? $\endgroup$ – upabove Sep 24 '11 at 9:47
  • $\begingroup$ also i'm asking about the practical aspects of this, do you use this conversion while performing analyses? $\endgroup$ – upabove Sep 24 '11 at 9:47
  • $\begingroup$ @ocram What is $n$? Is it the sample size? Or number of samples taken? Or both? $\endgroup$ – quirik Dec 25 '16 at 17:22
  • $\begingroup$ @quirik: The assumption is that a single sample is taken and that this sample is of size n $\endgroup$ – ocram Dec 25 '16 at 17:36
  • $\begingroup$ @ocram How do we then calculate expected value of variance if we have one sample? What am I missing? $\endgroup$ – quirik Dec 25 '16 at 17:39
6
$\begingroup$

This response clarifies ocram's answer. The key reason (and common misunderstanding) for $E[S^2] \neq \sigma^2$ is that $S^2$ uses the estimate $\bar{X}$ which is itself estimated from data.

If you work through the derivation, you will see that the variance of this estimate $E[(\bar{X}-\mu)^2]$ is exactly what gives the additional $-\frac{\sigma^2}{n}$ term

$\endgroup$
5
$\begingroup$

The explanation that @Ocram gave is great. To explain what he said in words: if we calculate $s^2$ by dividing just by $n$, (which is intuitive) our estimation of $s^2$ will be an underestimate. To compensate, we divide by $n-1$.

Here's an exercise: Make up a discrete probability with 2 outcomes, say $P(2) = .25$ and $P(6) = .75$. Find $\mu$ and $\sigma$ for this distribution. Calculate $\mu$ and $\sigma$ for the sample mean when $n = 3$. Calculate all possible samples of size $n =3$. Calculate $s^2$ over those samples, and apply appropriate frequencies.

Sometimes, you gotta get your hands dirty.

$\endgroup$
  • $\begingroup$ thanks for your help. A few questions: In your excercise: what kind of distribution are you referring to, Binomial? What do you mean make up a discrete probability? You mean calculate all the probabilities of 2 and 6 over different sample sizes? $\endgroup$ – upabove Sep 26 '11 at 20:46
1
$\begingroup$

Generally using "n" in the denominator gives smaller values than the population variance which is what we want to estimate. This especially happens if the small samples are taken. In the language of statistics, we say that the sample variance provides a “biased” estimate of the population variance and needs to be made "unbiased".

This video will answer each part of your question adequately.

https://www.youtube.com/watch?v=xslIhnquFoE

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.