PRML illustration of how bias arises in using maximum likelihood to determine the variance of a Gaussian

I'm reading PRML and I don't understand the picture. Could you please give some hints to understand the picture and why the MLE of variance in a Gaussian distribution is biased?

formula 1.55: $$ \mu_{MLE}=\frac{1}{N} \sum_{n=1}^N x_n $$ formula 1.56 $$ \sigma_{MLE}^2=\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu_{MLE})^2 $$

  • $\begingroup$ Please add the self-study tag. $\endgroup$ Feb 7, 2015 at 4:24
  • 4
    $\begingroup$ why for each graph, only one blue data point is visible to me? btw, while I was trying to edit the overflow of two subscripts in this post, the system requires "at least 6 characters"... embarrassing. $\endgroup$
    – Zhanxiong
    Feb 7, 2015 at 5:32
  • $\begingroup$ What do you really want to understand, the picture or why the MLE estimate of variance is biased? The former is very confusing but I can explain the latter. $\endgroup$ Feb 7, 2015 at 15:03
  • 2
    $\begingroup$ yeah, I found in new version each graph has two blue data, my pdf is old $\endgroup$
    – ningyuwhut
    Feb 7, 2015 at 15:13
  • $\begingroup$ @TrynnaDoStat sorry for my question is not clear. what I want to know is why the MLE estimate of variance is biased. and how this is expressed in this graph $\endgroup$
    – ningyuwhut
    Feb 7, 2015 at 15:15

2 Answers 2



The bias is "coming from" (not at all a technical term) the fact that $E[\bar{x}^2]$ is biased for $\mu^2$. The natural question is, "well, what's the intuition for why $E[\bar{x}^2]$ is biased for $\mu^2$"? The intuition is that in a non-squared sample mean, sometimes we miss the true value $\mu$ by over-estimating and sometimes by under-estimating. But, without squaring, the tendency to over-estimate and under-estimate will cancel each other out. However, when we square $\bar{x}$ the tendency to under-estimate (miss the true value of $\mu$ by a negative number) also gets squared and thus becomes positive. Thus, it no longer cancels out and there is a slight tendency to over-estimate.

If the intuition behind why $\bar{x}^2$ is biased for $\mu^2$ is still unclear, try to understand the intuition behind Jensen's inequality (good intuitive explanation here) and apply it to $E[\bar{x}^2]$.

Let's prove that the MLE of variance for an iid sample is biased. Then we will analytically verify our intuition.


Let $\hat{\sigma}^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2$.

We want to show $E[\hat{\sigma}^2] \neq \sigma^2$.

$$E[\hat{\sigma}^2] = E[\frac{1}{N}\sum_{n = 1}^N (x_n - \bar{x})^2] = \frac{1}{N}E[\sum_{n = 1}^N (x_n^2 - 2x_n\bar{x} + \bar{x}^2)] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2]$$

Using the fact that $\sum_{n = 1}^N x_n = N\bar{x}$ and $\sum_{n = 1}^N \bar{x}^2 = N\bar{x}^2$,

$$\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - \sum_{n = 1}^N 2x_n\bar{x} + \sum_{n = 1}^N \bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2 - 2N\bar{x}^2 + N\bar{x}^2]=\frac{1}{N}E[\sum_{n = 1}^N x_n^2 - N\bar{x}^2] = \frac{1}{N}E[\sum_{n = 1}^N x_n^2] - E[\bar{x}^2] = \frac{1}{N}\sum_{n = 1}^N E[x_n^2] - E[\bar{x}^2] \\= E[x_n^2] - E[\bar{x}^2]$$

With the last step following since due to $E[x_n^2]$ being equal across $n$ due to coming from the same distribution.

Now, recall the definition of variance that says $\sigma^2_x = E[x^2] - E[x]^2$. From here, we get the following

$$E[x_n^2] - E[\bar{x}^2] = \sigma^2_x + E[x_n]^2 - \sigma^2_\bar{x} - E[x_n]^2 = \sigma^2_x - \sigma^2_\bar{x} = \sigma^2_x - Var(\bar{x}) = \sigma^2_x - Var(\frac{1}{N}\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n)$$

Notice that we've appropriately squared the constant $\frac{1}{N}$ when taking it out of $Var()$. Pay special attention to that!

$$\sigma^2_x - \bigg(\frac{1}{N}\bigg)^2Var(\sum_{n = 1}^Nx_n) = \sigma^2_x - \bigg(\frac{1}{N}\bigg)^2N \sigma^2_x = \sigma^2_x - \frac{1}{N}\sigma^2_x = \frac{N-1}{N}\sigma^2_x$$

which is, of course, not equal to $\sigma_x^2$.

Analytically Verify our Intuition

We can somewhat verify the intuition by assuming we know the value of $\mu$ and plugging it into the above proof. Since we now know $\mu$, we no longer have the need to estimate $\mu^2$ and thus we never over-estimate it with $E[\bar{x}^2]$. Let's see that this "removes" the bias in $\hat{\sigma}^2$.

Let $\hat{\sigma}_\mu^2 = \frac{1}{N}\sum_{n = 1}^N (x_n - \mu)^2$.

From the above proof, let's pick up from $E[x_n^2] - E[\bar{x}^2]$ replacing $\bar{x}$ with the true value $\mu$.

$$E[x_n^2] - E[\mu^2] = E[x_n^2] - \mu^2 = \sigma^2_x + E[x_n]^2 - \mu^2= \sigma^2_x$$

which is unbiased!

  • 9
    $\begingroup$ +1 It may be worth remarking that your demonstration does not require that $X$ have a Gaussian distribution. (However, for other distributions the sample variance might not be the MLE for the variance parameter.) $\endgroup$
    – whuber
    Feb 7, 2015 at 18:04
  • 1
    $\begingroup$ Thanks for your explanation. I need some time to understand it.Besides, I found some error in the equations.can you verify it? Thanks! $\endgroup$
    – ningyuwhut
    Feb 12, 2015 at 16:55
  • $\begingroup$ @ whuber - Not sure why did you say "..demonstration does not require that $X$ have a Gaussian distribution.". We would not talk about ML solution of variance for every distribution, say a binomial distribution. So implicitly we're assuming X distribution has variance as one of the parameters. $\endgroup$
    – KGhatak
    Dec 11, 2019 at 8:24
  • $\begingroup$ Sorry, a bit off question. Like we call the one with division by N-1 as sample variance, how is the ML estimator $\hat{\sigma}_\mu^2$ called in the literature? $\endgroup$
    – Ashok
    May 19, 2021 at 1:06
  • $\begingroup$ what is the meaning of $x_n$ outside the sum over $n$? $\endgroup$
    – nuemlouno
    Mar 14, 2022 at 0:25

The maximum likelihood estimates of mean and variance for a Gaussian distribution are:

\begin{align*} \hat{\mu} &= \frac{1}{N} \sum_{i=1}^N x_i \\ \hat{\sigma}^2 &= \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})^2 \\ \end{align*}

Suppose the real mean and variance for the Gaussian distribution is $\mu$ and $\sigma^2$.

First, we show $\hat{\mu}$ is unbiased,

\begin{align*} \mathbb{E}[ \hat{\mu}] &= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N x_i \right ] \\ &= \frac{1}{N} \sum_{i=1}^N \mathbb{E} [x_i] \\ &= \frac{1}{N} \sum_{i=1}^N \mu \\ &= \mu \end{align*}

Next, we show $\hat{\sigma}^2$ is biased,

\begin{align} \mathbb{E}[ \hat{\sigma}^2] &= \mathbb{E} \left[ \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N (x_i - \hat{\mu})^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - \sum_{i=1}^N x_i \hat{\mu} - \sum_{i=1}^N \hat{\mu} x_i + \sum_{i=1}^N \hat{\mu}^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - N \hat{\mu}^2 - N\hat{\mu}^2 + N\hat{\mu}^2 \right ] \\ &= \frac{1}{N} \mathbb{E} \left[ \sum_{i=1}^N x_i^2 - N \hat{\mu}^2 \right ] \\ &= \mathbb{E}[x^2] - \mathbb{E}[\hat{\mu}^2] \\ &= \mathbb{E}[x^2] - \mathbb{E} \left[ \left( \frac{1}{N} \sum_i^N x_{i=1} \right ) \left( \frac{1}{N} \sum_{j=1}^N x_j \right ) \right] \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} \mathbb{E} \left[ \sum_{i=1}^N \sum_{j=1}^N x_i x_j \right] \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} \mathbb{E} \left[ \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i = j) x_i x_j + \sum_{i=1}^N \sum_{j=1}^N \mathbb{I}(i \ne j) x_i x_j \right] \\ &= \mathbb{E}[x^2] - \frac{1}{N^2} N \mathbb{E}[x^2] - \frac{1}{N^2} N(N - 1) \mu^2 \\ &= \frac{N - 1}{N} \mathbb{E}[x^2] - \frac{N-1}{N} \mu^2 \\ &= \frac{N - 1}{N} \left(\sigma^2 + \mu^2 \right ) - \frac{N-1}{N} \mu^2 \\ &= \frac{N - 1}{N} \sigma^2 \end{align}

So $\hat{\sigma}^2$ is an underestimation of $\sigma^2$. When $N=2$ as shown in the plot in the question, $\hat{\sigma}^2 = \frac{1}{2} \sigma^2$, which is a significant underestimation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.