# How to understand that MLE of Variance is biased in a Gaussian distribution?

I'm reading PRML and I don't understand the picture. Could you please give some hints to understand the picture and why the MLE of variance in a Gaussian distribution is biased?

formula 1.55: $$\mu_{MLE}=\frac{1}{N} \sum_{n=1}^N x_n$$ formula 1.56 $$\sigma_{MLE}^2=\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu_{MLE})^2$$

• Please add the self-study tag. Commented Feb 7, 2015 at 4:24
• why for each graph, only one blue data point is visible to me? btw, while I was trying to edit the overflow of two subscripts in this post, the system requires "at least 6 characters"... embarrassing. Commented Feb 7, 2015 at 5:32
• What do you really want to understand, the picture or why the MLE estimate of variance is biased? The former is very confusing but I can explain the latter. Commented Feb 7, 2015 at 15:03
• yeah, I found in new version each graph has two blue data, my pdf is old Commented Feb 7, 2015 at 15:13
• @TrynnaDoStat sorry for my question is not clear. what I want to know is why the MLE estimate of variance is biased. and how this is expressed in this graph Commented Feb 7, 2015 at 15:15

Intuition

The bias is "coming from" (not at all a technical term) the fact that $$E[\bar{x}^2]$$ is biased for $$\mu^2$$. The natural question is, "well, what's the intuition for why $$E[\bar{x}^2]$$ is biased for $$\mu^2$$"? The intuition is that in a non-squared sample mean, sometimes we miss the true value $$\mu$$ by over-estimating and sometimes by under-estimating. But, without squaring, the tendency to over-estimate and under-estimate will cancel each other out. However, when we square $$\bar{x}$$ the tendency to under-estimate (miss the true value of $$\mu$$ by a negative number) also gets squared and thus becomes positive. Thus, it no longer cancels out and there is a slight tendency to over-estimate.

If the intuition behind why $$\bar{x}^2$$ is biased for $$\mu^2$$ is still unclear, try to understand the intuition behind Jensen's inequality (good intuitive explanation here) and apply it to $$E[\bar{x}^2]$$.

Let's prove that the MLE of variance for an iid sample is biased. Then we will analytically verify our intuition.

Proof

Let $$\hat{\sigma}^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2$$.

We want to show $$E[\hat{\sigma}^2] \neq \sigma^2$$.

$$E[\hat{\sigma}^2] = E[\frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2] = \frac{1}{N}E[\sum_{n = 1}^N (x_n^2 - 2x_n\bar{x} + \bar{x}^2)] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2]$$

Using the fact that $$\sum_{n = 1}^N x_n = N\bar{x}$$ and $$\sum_{n = 1}^N \bar{x}^2 = N\bar{x}^2$$,

$$\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - 2N\bar{x}^2 + N\bar{x}^2]=\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - N\bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2] - E[\bar{x}^2] = \frac{1}{N}\sum_{n = 1}^N E[x_n^2] - E[\bar{x}^2] \\= E[x_n^2] - E[\bar{x}^2]$$

With the last step following since due to $$E[x_n^2]$$ being equal across $$n$$ due to coming from the same distribution.

Now, recall the definition of variance that says $$\sigma^2_x = E[x^2] - E[x]^2$$. From here, we get the following

$$E[x_n^2] - E[\bar{x}^2] = \sigma^2_x + E[x_n]^2 - \sigma^2_\bar{x} - E[x_n]^2 = \sigma^2_x - \sigma^2_\bar{x} = \sigma^2_x - Var(\bar{x}) = \sigma^2_x - Var(\frac{1}{N}\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n)$$

Notice that we've appropriately squared the constant $$\frac{1}{N}$$ when taking it out of $$Var()$$. Pay special attention to that!

$$\sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2N \sigma^2_x = \sigma^2_x - \frac{1}{N}\sigma^2_x = \frac{N-1}{N}\sigma^2_x$$

which is, of course, not equal to $$\sigma_x^2$$.

Analytically Verify our Intuition

We can somewhat verify the intuition by assuming we know the value of $$\mu$$ and plugging it into the above proof. Since we now know $$\mu$$, we no longer have the need to estimate $$\mu^2$$ and thus we never over-estimate it with $$E[\bar{x}^2]$$. Let's see that this "removes" the bias in $$\hat{\sigma}^2$$.

Let $$\hat{\sigma}_\mu^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \mu)^2$$.

From the above proof, let's pick up from $$E[x_n^2] - E[\bar{x}^2]$$ replacing $$\bar{x}$$ with the true value $$\mu$$.

$$E[x_n^2] - E[\mu^2] = E[x_n^2] - \mu^2 = \sigma^2_x + E[x_n]^2 - \mu^2= \sigma^2_x$$

which is unbiased!

• +1 It may be worth remarking that your demonstration does not require that $X$ have a Gaussian distribution. (However, for other distributions the sample variance might not be the MLE for the variance parameter.)
– whuber
Commented Feb 7, 2015 at 18:04
• Thanks for your explanation. I need some time to understand it.Besides, I found some error in the equations.can you verify it? Thanks! Commented Feb 12, 2015 at 16:55
• @ whuber - Not sure why did you say "..demonstration does not require that $X$ have a Gaussian distribution.". We would not talk about ML solution of variance for every distribution, say a binomial distribution. So implicitly we're assuming X distribution has variance as one of the parameters. Commented Dec 11, 2019 at 8:24
• Sorry, a bit off question. Like we call the one with division by N-1 as sample variance, how is the ML estimator $\hat{\sigma}_\mu^2$ called in the literature? Commented May 19, 2021 at 1:06
• what is the meaning of $x_n$ outside the sum over $n$? Commented Mar 14, 2022 at 0:25

The maximum likelihood estimates of mean and variance for a Gaussian distribution are:

\begin{align*} \hat{\mu} &= \frac{1}{N} \sum_{i=1}^N x_i \\ \hat{\sigma}^2 &= \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})^2 \\ \end{align*}

Suppose the real mean and variance for the Gaussian distribution is $$\mu$$ and $$\sigma^2$$.

First, we show $$\hat{\mu}$$ is unbiased,

\begin{align*} \mathbb{E}[ \hat{\mu}] &= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N x_i \right ] \\ &= \frac{1}{N} \sum_{i=1}^N \mathbb{E} [x_i] \\ &= \frac{1}{N} \sum_{i=1}^N \mu \\ &= \mu \end{align*}

Next, we show $$\hat{\sigma}^2$$ is biased,

\begin{align} \mathbb{E}[ \hat{\sigma}^2] &= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})^2 \right ] \tag{1} \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N (x_i - \hat{\mu})^2 \right ] \tag{2} \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - \sum_{i=1}^N x_i \hat{\mu} - \sum_{i=1}^N \hat{\mu} x_i + \sum_{i=1}^N \hat{\mu}^2 \right ] \tag{3} \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - N \hat{\mu}^2 - N\hat{\mu}^2 + N\hat{\mu}^2 \right ] \tag{4} \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - N \hat{\mu}^2 \right ] \tag{5} \\ &= \mathbb{E}[x^2] - \mathbb{E}[\hat{\mu}^2] \tag{6} \\ &= \mathbb{E}[x^2] - \mathbb{E} \left[ \left( \frac{1}{N} \sum_i^N x_{i=1} \right ) \left( \frac{1}{N} \sum_{j=1}^N x_j \right ) \right] \tag{7} \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} \mathbb{E} \left[ \sum_{i=1}^N \sum_{j=1}^N x_i x_j \right] \tag{8} \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} \mathbb{E} \left[ \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i = j) x_i x_j + \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i \ne j) x_i x_j \right] \tag{9} \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} N \mathbb{E}[x^2] - \frac{1}{N^2} N(N - 1) \mu^2 \tag{10} \\ &= \frac{N - 1}{N} \mathbb{E}[x^2] - \frac{N-1}{N} \mu^2 \tag{11} \\ &= \frac{N - 1}{N} \left(\sigma^2 + \mu^2 \right ) - \frac{N-1}{N} \mu^2 \tag{12} \\ &= \frac{N - 1}{N} \sigma^2 \tag{13} \end{align}

So $$\hat{\sigma}^2$$ is an underestimation of $$\sigma^2$$. When $$N=2$$ as shown in the plot in the question, $$\hat{\sigma}^2 = \frac{1}{2} \sigma^2$$, which is a significant underestimation.

Note, in the 10th equality, we used the fact that the sample is i.i.d., so

\begin{align} \mathbb{E}\left[ \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i \neq j)x_i x_j \right ] &= \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i \neq j) \mathbb{E}[x_i] \mathbb{E}[x_j] \\ &= (N^2 - N) \mu \mu \\ &= N(N-1)\mu^2 \end{align}