How to understand that MLE of Variance is biased in a Gaussian distribution?

Question

I'm reading PRML and I don't understand the picture. Could you please give some hints to understand the picture and why the MLE of variance in a Gaussian distribution is biased?

formula 1.55: $$ \mu_{MLE}=\frac{1}{N} \sum_{n=1}^N x_n $$ formula 1.56 $$ \sigma_{MLE}^2=\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu_{MLE})^2 $$

Answer 1

Intuition

The bias is "coming from" (not at all a technical term) the fact that $E[\bar{x}^2]$ is biased for $\mu^2$. The natural question is, "well, what's the intuition for why $E[\bar{x}^2]$ is biased for $\mu^2$"? The intuition is that in a non-squared sample mean, sometimes we miss the true value $\mu$ by over-estimating and sometimes by under-estimating. But, without squaring, the tendency to over-estimate and under-estimate will cancel each other out. However, when we square $\bar{x}$ the tendency to under-estimate (miss the true value of $\mu$ by a negative number) also gets squared and thus becomes positive. Thus, it no longer cancels out and there is a slight tendency to over-estimate.

If the intuition behind why $x^2$ is biased for $\mu^2$ is still unclear, try to understand the intuition behind Jensen's inequality (good intuitive explanation here) and apply it to $E[x^2]$.

Let's prove that the MLE of variance for an iid sample is biased. Then we will analytically verify our intuition.

Proof

Let $\hat{\sigma}^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2$.

We want to show $E[\hat{\sigma}^2] \neq \sigma^2$.

$$E[\hat{\sigma}^2] = E[\frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2] = \frac{1}{N}E[\sum_{n = 1}^N (x_n^2 - 2x_n\bar{x} + \bar{x}^2)] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2]$$

Using the fact that $\sum_{n = 1}^N x_n = N\bar{x}$ and $\sum_{n = 1}^N \bar{x}^2 = N\bar{x}^2$,

$$\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - 2N\bar{x}^2 + N\bar{x}^2]=\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - N\bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2] - E[\bar{x}^2] = \frac{1}{N}\sum_{n = 1}^N E[x_n^2] - E[\bar{x}^2] \\= E[x_n^2] - E[\bar{x}^2]$$

With the last step following since due to $E[x_n^2]$ being equal across $n$ due to coming from the same distribution.

Now, recall the definition of variance that says $\sigma^2_x = E[x^2] - E[x]^2$. From here, we get the following

$$E[x_n^2] - E[\bar{x}^2] = \sigma^2_x + E[x_n]^2 - \sigma^2_\bar{x} - E[x_n]^2 = \sigma^2_x - \sigma^2_\bar{x} = \sigma^2_x - Var(\bar{x}) = \sigma^2_x - Var(\frac{1}{N}\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n)$$

Notice that we've appropriately squared the constant $\frac{1}{N}$ when taking it out of $Var()$. Pay special attention to that!

$$\sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2N \sigma^2_x = \sigma^2_x - \frac{1}{N}\sigma^2_x = \frac{N-1}{N}\sigma^2_x$$

which is, of course, not equal to $\sigma_x^2$.

Analytically Verify our Intuition

We can somewhat verify the intuition by assuming we know the value of $\mu$ and plugging it into the above proof. Since we now know $\mu$, we no longer have the need to estimate $\mu^2$ and thus we never over-estimate it with $E[\bar{x}^2]$. Let's see that this "removes" the bias in $\hat{\sigma}^2$.

Let $\hat{\sigma}_\mu^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \mu)^2$.

From the above proof, let's pick up from $E[x_n^2] - E[\bar{x}^2]$ replacing $\bar{x}$ with the true value $\mu$.

$$E[x_n^2] - E[\mu^2] = E[x_n^2] - \mu^2 = \sigma^2_x + E[x_n]^2 - \mu^2= \sigma^2_x$$

which is unbiased!

Answer 2

The maximum likelihood estimates of mean and variance for a Gaussian distribution are:

\begin{align*} \hat{\mu} &= \frac{1}{N} \sum_{i=1}^N x_i \\ \hat{\sigma}^2 &= \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})^2 \\ \end{align*}

Suppose the real mean and variance for the Gaussian distribution is $\mu$ and $\sigma^2$.

First, we show $\hat{\mu}$ is unbiased,

\begin{align*} \mathbb{E}[ \hat{\mu}] &= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N x_i \right ] \\ &= \frac{1}{N} \sum_{i=1}^N \mathbb{E} [x_i] \\ &= \frac{1}{N} \sum_{i=1}^N \mu \\ &= \mu \end{align*}

Next, we show $\hat{\sigma}^2$ is biased,

\begin{align} \mathbb{E}[ \hat{\sigma}^2] &= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N (x_i - \hat{\mu})^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - \sum_{i=1}^N x_i \hat{\mu} - \sum_{i=1}^N \hat{\mu} x_i + \sum_{i=1}^N \hat{\mu}^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - N \hat{\mu}^2 - N\hat{\mu}^2 + N\hat{\mu}^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - N \hat{\mu}^2 \right ] \\ &= \mathbb{E}[x^2] - \mathbb{E}[\hat{\mu}^2] \\ &= \mathbb{E}[x^2] - \mathbb{E} \left[ \left( \frac{1}{N} \sum_i^N x_{i=1} \right ) \left( \frac{1}{N} \sum_{j=1}^N x_j \right ) \right] \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} \mathbb{E} \left[ \sum_{i=1}^N \sum_{j=1}^N x_i x_j \right] \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} \mathbb{E} \left[ \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i = j) x_i x_j + \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i \ne j) x_i x_j \right] \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} N \mathbb{E}[x^2] - \frac{1}{N^2} N(N - 1) \mu^2 \\ &= \frac{N - 1}{N} \mathbb{E}[x^2] - \frac{N-1}{N} \mu^2 \\ &= \frac{N - 1}{N} \left(\sigma^2 + \mu^2 \right ) - \frac{N-1}{N} \mu^2 \\ &= \frac{N - 1}{N} \sigma^2 \end{align}

So $\hat{\sigma}^2$ is an underestimation of $\sigma^2$. When $N=2$ as shown in the plot in the question, $\hat{\sigma}^2 = \frac{1}{2} \sigma^2$, which is a significant underestimation.