# How to understand that MLE of Variance is biased in a Gaussian distribution? I'm reading PRML and I don't understand the picture. Could you please give some hints to understand the picture and why the MLE of variance in a Gaussian distribution is biased?

formula 1.55: $$\mu_{MLE}=\frac{1}{N} \sum_{n=1}^N x_n$$ formula 1.56 $$\sigma_{MLE}^2=\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu_{MLE})^2$$

• why for each graph, only one blue data point is visible to me? btw, while I was trying to edit the overflow of two subscripts in this post, the system requires "at least 6 characters"... embarrassing. Feb 7, 2015 at 5:32
• What do you really want to understand, the picture or why the MLE estimate of variance is biased? The former is very confusing but I can explain the latter. Feb 7, 2015 at 15:03
• yeah, I found in new version each graph has two blue data, my pdf is old Feb 7, 2015 at 15:13
• @TrynnaDoStat sorry for my question is not clear. what I want to know is why the MLE estimate of variance is biased. and how this is expressed in this graph Feb 7, 2015 at 15:15

Intuition

The bias is "coming from" (not at all a technical term) the fact that $E[\bar{x}^2]$ is biased for $\mu^2$. The natural question is, "well, what's the intuition for why $E[\bar{x}^2]$ is biased for $\mu^2$"? The intuition is that in a non-squared sample mean, sometimes we miss the true value $\mu$ by over-estimating and sometimes by under-estimating. But, without squaring, the tendency to over-estimate and under-estimate will cancel each other out. However, when we square $\bar{x}$ the tendency to under-estimate (miss the true value of $\mu$ by a negative number) also gets squared and thus becomes positive. Thus, it no longer cancels out and there is a slight tendency to over-estimate.

If the intuition behind why $x^2$ is biased for $\mu^2$ is still unclear, try to understand the intuition behind Jensen's inequality (good intuitive explanation here) and apply it to $E[x^2]$.

Let's prove that the MLE of variance for an iid sample is biased. Then we will analytically verify our intuition.

Proof

Let $\hat{\sigma}^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2$.

We want to show $E[\hat{\sigma}^2] \neq \sigma^2$.

$$E[\hat{\sigma}^2] = E[\frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2] = \frac{1}{N}E[\sum_{n = 1}^N (x_n^2 - 2x_n\bar{x} + \bar{x}^2)] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2]$$

Using the fact that $\sum_{n = 1}^N x_n = N\bar{x}$ and $\sum_{n = 1}^N \bar{x}^2 = N\bar{x}^2$,

$$\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - 2N\bar{x}^2 + N\bar{x}^2]=\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - N\bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2] - E[\bar{x}^2] = \frac{1}{N}\sum_{n = 1}^N E[x_n^2] - E[\bar{x}^2] \\= E[x_n^2] - E[\bar{x}^2]$$

With the last step following since due to $E[x_n^2]$ being equal across $n$ due to coming from the same distribution.

Now, recall the definition of variance that says $\sigma^2_x = E[x^2] - E[x]^2$. From here, we get the following

$$E[x_n^2] - E[\bar{x}^2] = \sigma^2_x + E[x_n]^2 - \sigma^2_\bar{x} - E[x_n]^2 = \sigma^2_x - \sigma^2_\bar{x} = \sigma^2_x - Var(\bar{x}) = \sigma^2_x - Var(\frac{1}{N}\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n)$$

Notice that we've appropriately squared the constant $\frac{1}{N}$ when taking it out of $Var()$. Pay special attention to that!

$$\sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2N \sigma^2_x = \sigma^2_x - \frac{1}{N}\sigma^2_x = \frac{N-1}{N}\sigma^2_x$$

which is, of course, not equal to $\sigma_x^2$.

Analytically Verify our Intuition

We can somewhat verify the intuition by assuming we know the value of $\mu$ and plugging it into the above proof. Since we now know $\mu$, we no longer have the need to estimate $\mu^2$ and thus we never over-estimate it with $E[\bar{x}^2]$. Let's see that this "removes" the bias in $\hat{\sigma}^2$.

Let $\hat{\sigma}_\mu^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \mu)^2$.

From the above proof, let's pick up from $E[x_n^2] - E[\bar{x}^2]$ replacing $\bar{x}$ with the true value $\mu$.

$$E[x_n^2] - E[\mu^2] = E[x_n^2] - \mu^2 = \sigma^2_x + E[x_n]^2 - \mu^2= \sigma^2_x$$

which is unbiased!

• +1 It may be worth remarking that your demonstration does not require that $X$ have a Gaussian distribution. (However, for other distributions the sample variance might not be the MLE for the variance parameter.)
– whuber
Feb 7, 2015 at 18:04
• Thanks for your explanation. I need some time to understand it.Besides, I found some error in the equations.can you verify it? Thanks! Feb 12, 2015 at 16:55
• @ whuber - Not sure why did you say "..demonstration does not require that $X$ have a Gaussian distribution.". We would not talk about ML solution of variance for every distribution, say a binomial distribution. So implicitly we're assuming X distribution has variance as one of the parameters. Dec 11, 2019 at 8:24
• Sorry, a bit off question. Like we call the one with division by N-1 as sample variance, how is the ML estimator $\hat{\sigma}_\mu^2$ called in the literature? May 19, 2021 at 1:06
• what is the meaning of $x_n$ outside the sum over $n$? Mar 14, 2022 at 0:25

The maximum likelihood estimates of mean and variance for a Gaussian distribution are:

\begin{align*} \hat{\mu} &= \frac{1}{N} \sum_{i=1}^N x_i \\ \hat{\sigma}^2 &= \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})^2 \\ \end{align*}

Suppose the real mean and variance for the Gaussian distribution is $$\mu$$ and $$\sigma^2$$.

First, we show $$\hat{\mu}$$ is unbiased,

\begin{align*} \mathbb{E}[ \hat{\mu}] &= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N x_i \right ] \\ &= \frac{1}{N} \sum_{i=1}^N \mathbb{E} [x_i] \\ &= \frac{1}{N} \sum_{i=1}^N \mu \\ &= \mu \end{align*}

Next, we show $$\hat{\sigma}^2$$ is biased,

\begin{align} \mathbb{E}[ \hat{\sigma}^2] &= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N (x_i - \hat{\mu})^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - \sum_{i=1}^N x_i \hat{\mu} - \sum_{i=1}^N \hat{\mu} x_i + \sum_{i=1}^N \hat{\mu}^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - N \hat{\mu}^2 - N\hat{\mu}^2 + N\hat{\mu}^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - N \hat{\mu}^2 \right ] \\ &= \mathbb{E}[x^2] - \mathbb{E}[\hat{\mu}^2] \\ &= \mathbb{E}[x^2] - \mathbb{E} \left[ \left( \frac{1}{N} \sum_i^N x_{i=1} \right ) \left( \frac{1}{N} \sum_{j=1}^N x_j \right ) \right] \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} \mathbb{E} \left[ \sum_{i=1}^N \sum_{j=1}^N x_i x_j \right] \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} \mathbb{E} \left[ \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i = j) x_i x_j + \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i \ne j) x_i x_j \right] \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} N \mathbb{E}[x^2] - \frac{1}{N^2} N(N - 1) \mu^2 \\ &= \frac{N - 1}{N} \mathbb{E}[x^2] - \frac{N-1}{N} \mu^2 \\ &= \frac{N - 1}{N} \left(\sigma^2 + \mu^2 \right ) - \frac{N-1}{N} \mu^2 \\ &= \frac{N - 1}{N} \sigma^2 \end{align}

So $$\hat{\sigma}^2$$ is an underestimation of $$\sigma^2$$. When $$N=2$$ as shown in the plot in the question, $$\hat{\sigma}^2 = \frac{1}{2} \sigma^2$$, which is a significant underestimation.