$$ \hat{\sigma}^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2 $$ When proving the variance is underestimated using MLE, the following two procedures are generating different results. $$ E[\hat{\sigma}^2] = E[\frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2] \\= \frac{1}{N}\sum_{n=1}^N E[(x_n^2 - 2x_n\bar{x} + \bar{x}^2)] \\= \frac{1}{N}\sum_{n=1}^N E[x_n^2 - 2x_n\bar{x} + \bar{x}^2] \\= \frac{1}{N}\sum_{n=1}^N (E[x_n^2]-2\bar{x}E[x_n]+E[\bar{x}^2])\\= \frac{1}{N}\sum_{n=1}^N (E[x_n^2]-2\bar{x}^2+E[\bar{x}^2])\\=E[x_n^2]-2\bar{x}^2+E[\bar{x}^2] $$

$$ E[\hat{\sigma}^2] = E[\frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2] \\= \frac{1}{N}E[\sum_{n = 1}^N (x_n^2 - 2x_n\bar{x} + \bar{x}^2)] \\= \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2] \\= \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - 2N\bar{x}^2 + N\bar{x}^2]\\=\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - N\bar{x}^2] \\= \frac{1}{N}E[\sum_{n = 1}^N x_n^2] - E[\bar{x}^2] \\= \frac{1}{N}\sum_{n = 1}^N E[x_n^2] - E[\bar{x}^2] \\= E[x_n^2] - E[\bar{x}^2] $$

Can anyone point out which step is wrong in the first procedure?

  • 4
    $\begingroup$ Just add some comments here, you should use capital letters here and $E(X_n)=\mu$ not $\bar{X}$ $\endgroup$ – Deep North Sep 28 '17 at 8:55
  • 2
    $\begingroup$ Also, worth pointing out that $\mathbb{E}[\hat \sigma^2] = \sigma^2(1-1/N)$ underestimates the true variance. Hence, you divide by $N-1$ instead of $N$ to get the unbiased estimator. $\endgroup$ – P.Windridge Sep 28 '17 at 9:40
  • 1
    $\begingroup$ I suggest you carefully try to compute first $\mathbb{E}[X_i \bar X]$ and then $\mathbb{E}[\bar X^2]$, remembering that if $i=j$ then $\mathbb{E}[X_i X_j] = \mathbb{E}[X_i^2] \neq \mu^2$! $\endgroup$ – P.Windridge Sep 28 '17 at 9:47
  • $\begingroup$ @P.Windridge Thanks for pointing out. I made a typo on that in the original post. Corrected now $\endgroup$ – rifle123 Sep 28 '17 at 15:43

Your problem stems from two facts. First, look at the last equality of the first aproach. You have wrongly assumed that the inequality below is an equality.

$$E[x^2_n] \neq E[\bar x^2]$$

Second, as pointed out in the comments, you should probably check your notion of expected value. $\bar x$ is an estimator, and $E[x] \neq \bar x$.

To avoid problems in the future, consider that $E[X]$ is the mean and $\bar x$ is the empirical mean. While $\bar x$ is a function of your data, $E[X]$ depends on the distribution of $X$, which you don't really know if you just have a bunch of observations.

  • $\begingroup$ Thanks for correcting. I made a typo on the first issue, corrected it in the post. The second point caught my mistake. One more question, when proving this, why \bar{x}^2 could not be taken out from expectation? $\endgroup$ – rifle123 Sep 28 '17 at 16:47
  • $\begingroup$ Because it is a random variable and not a constant (only constants can be taken out of the expectation E() ) $\endgroup$ – Zahava Kor Sep 28 '17 at 19:18
  • $\begingroup$ @rifle123 Exactly. Consider that the expected value is simply an integral or sum, weighted by the probability of observing each value of the variable. That is why constants can be factored out. However, estimators are in general different depending on the set of outcomes we observed. $\endgroup$ – broncoAbierto Sep 29 '17 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.