12
$\begingroup$

PRML illustration of how bias arises in using maximum likelihood to determine the variance of a Gaussian

I'm reading PRML and I don't understand the picture. Could you please give some hints to understand the picture and why the MLE of variance in a Gaussian distribution is biased?

formula 1.55: $$ \mu_{MLE}=\frac{1}{N} \sum_{n=1}^N x_n $$ formula 1.56 $$ \sigma_{MLE}^2=\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu_{MLE})^2 $$

$\endgroup$
  • $\begingroup$ Please add the self-study tag. $\endgroup$ – StatsStudent Feb 7 '15 at 4:24
  • 2
    $\begingroup$ why for each graph, only one blue data point is visible to me? btw, while I was trying to edit the overflow of two subscripts in this post, the system requires "at least 6 characters"... embarrassing. $\endgroup$ – Zhanxiong Feb 7 '15 at 5:32
  • $\begingroup$ What do you really want to understand, the picture or why the MLE estimate of variance is biased? The former is very confusing but I can explain the latter. $\endgroup$ – TrynnaDoStat Feb 7 '15 at 15:03
  • $\begingroup$ yeah, I found in new version each graph has two blue data, my pdf is old $\endgroup$ – ningyuwhut Feb 7 '15 at 15:13
  • $\begingroup$ @TrynnaDoStat sorry for my question is not clear. what I want to know is why the MLE estimate of variance is biased. and how this is expressed in this graph $\endgroup$ – ningyuwhut Feb 7 '15 at 15:15
23
$\begingroup$

Intuition

The bias is "coming from" (not at all a technical term) the fact that $E[\bar{x}^2]$ is biased for $\mu^2$. The natural question is, "well, what's the intuition for why $E[\bar{x}^2]$ is biased for $\mu^2$"? The intuition is that in a non-squared sample mean, sometimes we miss the true value $\mu$ by over-estimating and sometimes by under-estimating. But, without squaring, the tendency to over-estimate and under-estimate will cancel each other out. However, when we square $\bar{x}$ the tendency to under-estimate (miss the true value of $\mu$ by a negative number) also gets squared and thus becomes positive. Thus, it no longer cancels out and there is a slight tendency to over-estimate.

If the intuition behind why $x^2$ is biased for $\mu^2$ is still unclear, try to understand the intuition behind Jensen's inequality (good intuitive explanation here) and apply it to $E[x^2]$.

Let's prove that the MLE of variance for an iid sample is biased. Then we will analytically verify our intuition.

Proof

Let $\hat{\sigma}^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2$.

We want to show $E[\hat{\sigma}^2] \neq \sigma^2$.

$$E[\hat{\sigma}^2] = E[\frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2] = \frac{1}{N}E[\sum_{n = 1}^N (x_n^2 - 2x_n\bar{x} + \bar{x}^2)] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2]$$

Using the fact that $\sum_{n = 1}^N x_n = N\bar{x}$ and $\sum_{n = 1}^N \bar{x}^2 = N\bar{x}^2$,

$$\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - 2N\bar{x}^2 + N\bar{x}^2]=\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - N\bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2] - E[\bar{x}^2] = \frac{1}{N}\sum_{n = 1}^N E[x_n^2] - E[\bar{x}^2] \\= E[x_n^2] - E[\bar{x}^2]$$

With the last step following since due to $E[x_n^2]$ being equal across $n$ due to coming from the same distribution.

Now, recall the definition of variance that says $\sigma^2_x = E[x^2] - E[x]^2$. From here, we get the following

$$E[x_n^2] - E[\bar{x}^2] = \sigma^2_x + E[x_n]^2 - \sigma^2_\bar{x} - E[x_n]^2 = \sigma^2_x - \sigma^2_\bar{x} = \sigma^2_x - Var(\bar{x}) = \sigma^2_x - Var(\frac{1}{N}\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n)$$

Notice that we've appropriately squared the constant $\frac{1}{N}$ when taking it out of $Var()$. Pay special attention to that!

$$\sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2N \sigma^2_x = \sigma^2_x - \frac{1}{N}\sigma^2_x = \frac{N-1}{N}\sigma^2_x$$

which is, of course, not equal to $\sigma_x^2$.

Analytically Verify our Intuition

We can somewhat verify the intuition by assuming we know the value of $\mu$ and plugging it into the above proof. Since we now know $\mu$, we no longer have the need to estimate $\mu^2$ and thus we never over-estimate it with $E[\bar{x}^2]$. Let's see that this "removes" the bias in $\hat{\sigma}^2$.

Let $\hat{\sigma}_\mu^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \mu)^2$.

From the above proof, let's pick up from $E[x_n^2] - E[\bar{x}^2]$ replacing $\bar{x}$ with the true value $\mu$.

$$E[x_n^2] - E[\mu^2] = E[x_n^2] - \mu^2 = \sigma^2_x + E[x_n]^2 - \mu^2= \sigma^2_x$$

which is unbiased!

$\endgroup$
  • 3
    $\begingroup$ +1 It may be worth remarking that your demonstration does not require that $X$ have a Gaussian distribution. (However, for other distributions the sample variance might not be the MLE for the variance parameter.) $\endgroup$ – whuber Feb 7 '15 at 18:04
  • 1
    $\begingroup$ Thanks for your explanation. I need some time to understand it.Besides, I found some error in the equations.can you verify it? Thanks! $\endgroup$ – ningyuwhut Feb 12 '15 at 16:55
  • $\begingroup$ @ whuber - Not sure why did you say "..demonstration does not require that $X$ have a Gaussian distribution.". We would not talk about ML solution of variance for every distribution, say a binomial distribution. So implicitly we're assuming X distribution has variance as one of the parameters. $\endgroup$ – KGhatak Dec 11 '19 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.