How to understand that MLE of Variance is biased in a Gaussian distribution? 
I'm reading PRML and I don't understand the picture. Could you please give some hints to understand the picture and why the MLE of variance in a Gaussian distribution is biased?
formula 1.55:
$$
\mu_{MLE}=\frac{1}{N} \sum_{n=1}^N x_n
$$
formula 1.56
$$
\sigma_{MLE}^2=\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu_{MLE})^2
$$
 A: Intuition
The bias is "coming from" (not at all a technical term) the fact that $E[\bar{x}^2]$ is biased for $\mu^2$. The natural question is, "well, what's the intuition for why $E[\bar{x}^2]$ is biased for $\mu^2$"? The intuition is that in a non-squared sample mean, sometimes we miss the true value $\mu$ by over-estimating and sometimes by under-estimating. But, without squaring, the tendency to over-estimate and under-estimate will cancel each other out. However, when we square $\bar{x}$ the tendency to under-estimate (miss the true value of $\mu$ by a negative number) also gets squared and thus becomes positive. Thus, it no longer cancels out and there is a slight tendency to over-estimate.
If the intuition behind why $x^2$ is biased for $\mu^2$ is still unclear, try to understand the intuition behind Jensen's inequality (good intuitive explanation here) and apply it to $E[x^2]$.
Let's prove that the MLE of variance for an iid sample is biased. Then we will analytically verify our intuition.
Proof
Let $\hat{\sigma}^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2$.
We want to show $E[\hat{\sigma}^2] \neq \sigma^2$.
$$E[\hat{\sigma}^2] = E[\frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2] = \frac{1}{N}E[\sum_{n = 1}^N (x_n^2 - 2x_n\bar{x} + \bar{x}^2)] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2]$$
Using the fact that $\sum_{n = 1}^N x_n = N\bar{x}$ and $\sum_{n = 1}^N \bar{x}^2 = N\bar{x}^2$,
$$\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - 2N\bar{x}^2 + N\bar{x}^2]=\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - N\bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2] - E[\bar{x}^2] = \frac{1}{N}\sum_{n = 1}^N E[x_n^2] - E[\bar{x}^2] \\= E[x_n^2] - E[\bar{x}^2]$$
With the last step following since due to $E[x_n^2]$ being equal across $n$ due to coming from the same distribution.
Now, recall the definition of variance that says $\sigma^2_x = E[x^2] - E[x]^2$. From here, we get the following
$$E[x_n^2] - E[\bar{x}^2] = \sigma^2_x + E[x_n]^2 - \sigma^2_\bar{x} - E[x_n]^2 = \sigma^2_x - \sigma^2_\bar{x} = \sigma^2_x - Var(\bar{x}) = \sigma^2_x - Var(\frac{1}{N}\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n)$$
Notice that we've appropriately squared the constant $\frac{1}{N}$ when taking it out of $Var()$. Pay special attention to that!
$$\sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2N \sigma^2_x = \sigma^2_x - \frac{1}{N}\sigma^2_x = \frac{N-1}{N}\sigma^2_x$$
which is, of course, not equal to $\sigma_x^2$.
Analytically Verify our Intuition
We can somewhat verify the intuition by assuming we know the value of $\mu$ and plugging it into the above proof. Since we now know $\mu$, we no longer have the need to estimate $\mu^2$ and thus we never over-estimate it with $E[\bar{x}^2]$. Let's see that this "removes" the bias in $\hat{\sigma}^2$.
Let $\hat{\sigma}_\mu^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \mu)^2$.
From the above proof, let's pick up from $E[x_n^2] - E[\bar{x}^2]$ replacing $\bar{x}$ with the true value $\mu$.
$$E[x_n^2] - E[\mu^2] = E[x_n^2] - \mu^2 = \sigma^2_x + E[x_n]^2 - \mu^2= \sigma^2_x$$
which is unbiased!
A: The maximum likelihood estimates of mean and variance for a Gaussian distribution are:
\begin{align*}
\hat{\mu} &= \frac{1}{N} \sum_{i=1}^N x_i \\
\hat{\sigma}^2 &= \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})^2 \\
\end{align*}
Suppose the real mean and variance for the Gaussian distribution is $\mu$ and $\sigma^2$.
First, we show $\hat{\mu}$ is unbiased,
\begin{align*}
\mathbb{E}[ \hat{\mu}] 
&= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N x_i \right ] \\
&= \frac{1}{N} \sum_{i=1}^N \mathbb{E} [x_i] \\
&= \frac{1}{N} \sum_{i=1}^N \mu \\
&= \mu
\end{align*}
Next, we show $\hat{\sigma}^2$ is biased,
\begin{align}
\mathbb{E}[ \hat{\sigma}^2] 
&= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})^2 \right ] \\
&= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N (x_i - \hat{\mu})^2 \right ] \\
&= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - \sum_{i=1}^N x_i \hat{\mu} - \sum_{i=1}^N \hat{\mu} x_i + \sum_{i=1}^N \hat{\mu}^2 \right ] \\
&= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - N \hat{\mu}^2 - N\hat{\mu}^2 + N\hat{\mu}^2 \right ] \\
&= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - N \hat{\mu}^2 \right ] \\
&= \mathbb{E}[x^2] - \mathbb{E}[\hat{\mu}^2] \\
&= \mathbb{E}[x^2] - \mathbb{E} \left[ \left( \frac{1}{N} \sum_i^N x_{i=1} \right ) \left( \frac{1}{N} \sum_{j=1}^N x_j \right )  \right] \\
&= \mathbb{E}[x^2] - \frac{1}{N^2} \mathbb{E} \left[ \sum_{i=1}^N \sum_{j=1}^N x_i x_j \right] \\
&= \mathbb{E}[x^2] - \frac{1}{N^2} \mathbb{E} \left[ \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i = j) x_i x_j + \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i \ne j) x_i x_j \right] \\
&= \mathbb{E}[x^2] - \frac{1}{N^2} N \mathbb{E}[x^2] - \frac{1}{N^2} N(N - 1) \mu^2 \\
&= \frac{N - 1}{N} \mathbb{E}[x^2] - \frac{N-1}{N} \mu^2 \\
&= \frac{N - 1}{N} \left(\sigma^2 + \mu^2 \right ) - \frac{N-1}{N} \mu^2 \\
&= \frac{N - 1}{N} \sigma^2
\end{align}
So $\hat{\sigma}^2$ is an underestimation of $\sigma^2$. When $N=2$ as shown in the plot in the question, $\hat{\sigma}^2 = \frac{1}{2} \sigma^2$, which is a significant underestimation.
