# How to understand that MLE of Variance is biased in a Gaussian distribution? I'm reading PRML and I don't understand the picture. Could you please give some hints to understand the picture and why the MLE of variance in a Gaussian distribution is biased?

formula 1.55: $$\mu_{MLE}=\frac{1}{N} \sum_{n=1}^N x_n$$ formula 1.56 $$\sigma_{MLE}^2=\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu_{MLE})^2$$

• Please add the self-study tag. – StatsStudent Feb 7 '15 at 4:24
• why for each graph, only one blue data point is visible to me? btw, while I was trying to edit the overflow of two subscripts in this post, the system requires "at least 6 characters"... embarrassing. – Zhanxiong Feb 7 '15 at 5:32
• What do you really want to understand, the picture or why the MLE estimate of variance is biased? The former is very confusing but I can explain the latter. – TrynnaDoStat Feb 7 '15 at 15:03
• yeah, I found in new version each graph has two blue data, my pdf is old – ningyuwhut Feb 7 '15 at 15:13
• @TrynnaDoStat sorry for my question is not clear. what I want to know is why the MLE estimate of variance is biased. and how this is expressed in this graph – ningyuwhut Feb 7 '15 at 15:15

Intuition

The bias is "coming from" (not at all a technical term) the fact that $E[\bar{x}^2]$ is biased for $\mu^2$. The natural question is, "well, what's the intuition for why $E[\bar{x}^2]$ is biased for $\mu^2$"? The intuition is that in a non-squared sample mean, sometimes we miss the true value $\mu$ by over-estimating and sometimes by under-estimating. But, without squaring, the tendency to over-estimate and under-estimate will cancel each other out. However, when we square $\bar{x}$ the tendency to under-estimate (miss the true value of $\mu$ by a negative number) also gets squared and thus becomes positive. Thus, it no longer cancels out and there is a slight tendency to over-estimate.

If the intuition behind why $x^2$ is biased for $\mu^2$ is still unclear, try to understand the intuition behind Jensen's inequality (good intuitive explanation here) and apply it to $E[x^2]$.

Let's prove that the MLE of variance for an iid sample is biased. Then we will analytically verify our intuition.

Proof

Let $\hat{\sigma}^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2$.

We want to show $E[\hat{\sigma}^2] \neq \sigma^2$.

$$E[\hat{\sigma}^2] = E[\frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2] = \frac{1}{N}E[\sum_{n = 1}^N (x_n^2 - 2x_n\bar{x} + \bar{x}^2)] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2]$$

Using the fact that $\sum_{n = 1}^N x_n = N\bar{x}$ and $\sum_{n = 1}^N \bar{x}^2 = N\bar{x}^2$,

$$\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - 2N\bar{x}^2 + N\bar{x}^2]=\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - N\bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2] - E[\bar{x}^2] = \frac{1}{N}\sum_{n = 1}^N E[x_n^2] - E[\bar{x}^2] \\= E[x_n^2] - E[\bar{x}^2]$$

With the last step following since due to $E[x_n^2]$ being equal across $n$ due to coming from the same distribution.

Now, recall the definition of variance that says $\sigma^2_x = E[x^2] - E[x]^2$. From here, we get the following

$$E[x_n^2] - E[\bar{x}^2] = \sigma^2_x + E[x_n]^2 - \sigma^2_\bar{x} - E[x_n]^2 = \sigma^2_x - \sigma^2_\bar{x} = \sigma^2_x - Var(\bar{x}) = \sigma^2_x - Var(\frac{1}{N}\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n)$$

Notice that we've appropriately squared the constant $\frac{1}{N}$ when taking it out of $Var()$. Pay special attention to that!

$$\sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2N \sigma^2_x = \sigma^2_x - \frac{1}{N}\sigma^2_x = \frac{N-1}{N}\sigma^2_x$$

which is, of course, not equal to $\sigma_x^2$.

Analytically Verify our Intuition

We can somewhat verify the intuition by assuming we know the value of $\mu$ and plugging it into the above proof. Since we now know $\mu$, we no longer have the need to estimate $\mu^2$ and thus we never over-estimate it with $E[\bar{x}^2]$. Let's see that this "removes" the bias in $\hat{\sigma}^2$.

Let $\hat{\sigma}_\mu^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \mu)^2$.

From the above proof, let's pick up from $E[x_n^2] - E[\bar{x}^2]$ replacing $\bar{x}$ with the true value $\mu$.

$$E[x_n^2] - E[\mu^2] = E[x_n^2] - \mu^2 = \sigma^2_x + E[x_n]^2 - \mu^2= \sigma^2_x$$

which is unbiased!

• +1 It may be worth remarking that your demonstration does not require that $X$ have a Gaussian distribution. (However, for other distributions the sample variance might not be the MLE for the variance parameter.) – whuber Feb 7 '15 at 18:04
• Thanks for your explanation. I need some time to understand it.Besides, I found some error in the equations.can you verify it? Thanks! – ningyuwhut Feb 12 '15 at 16:55
• @ whuber - Not sure why did you say "..demonstration does not require that $X$ have a Gaussian distribution.". We would not talk about ML solution of variance for every distribution, say a binomial distribution. So implicitly we're assuming X distribution has variance as one of the parameters. – KGhatak Dec 11 '19 at 8:24