# How to calculate the Expected maximum likelihood variance and mean for gaussian?

I am familiar with the Maximum Likelihood Estimation and Gaussian distribution. I have to big question from Bishop pattern recognition 2006 book.

1. As it is written on page 27, It says that the maximum likelihood underestimates variance.
2. It shows that the correct value for variance which equals to:

$$E[\sigma_{ml}^{2}] = (\frac{N - 1}{N})\sigma^2$$

I have know idea how it is calculated and didn't get its point and intuition. Would someone tell me what the above equation really is and talk about its application?

The ML (maximum likelihood) estimate for the variance is $$\sigma^2_{ml}=\frac{1}{n}\sum_{i=1}^n (X_i-\bar X)^2$$ where $$\bar X=\frac{1}{n}\sum_{j=1}^n X_j$$. Given formula is the expected value of this expression, assuming $$E[X_i]=\mu$$ and $$\operatorname{var}(X_i)=\sigma^2$$ and independent samples. It's a bit tedious but you can find the derivation here.

It underestimates the variance because $$\frac{N-1}{N}\sigma^2<\sigma^2$$.

This is a late answer, but I was just trying to show the same thing, so here it is. It's quite similar to the derivation suggested by the other answer (Wikipedia), but I found this easier to understand.

As already answered, what this equation is basically saying is that the maximum-likelihood value of the variance is, on average, less than the actual variance of the distribution that the data points $$x_1, ..., x_N$$ are drawn from. As far as I understand, Bishop is essentially giving an example of the kind of "issues" that stem from taking a maximum-likelihood approach in this context.

As for showing the relation, here is how I went about it:

First note that $$\mathbb{E}[\mu_{\text{ML}}] = \mathbb{E}\left[\frac{1}{N}\sum_{n=1}^{N}x_n\right] = \frac{1}{N}\sum_{n=1}^{n}\mathbb{E}[x_n] = \mu$$ This is because each of the $$x_n$$ are drawn from the same Gaussian distribution with mean $$\mu$$ and variance $$\sigma^2$$.

Similarly, $$\begin{equation*} \begin{split} \mathbb{E}[\mu_{\text{ML}}^2] = \frac{1}{N^2}\sum_{n, m}\mathbb{E}[x_nx_m] = \frac{1}{N^2}\sum_{n\neq m}\mathbb{E}[x_nx_m] + \frac{1}{N^2}\sum_{n=m}\mathbb{E}[x_nx_m] \\ = \frac{1}{N^2}\sum_{n\neq m}\mathbb{E}[x_n]\mathbb{E}[x_m] + \frac{\mu^2+\sigma^2}{N} = \frac{N^2 - N}{N^2}\mu^2 + \frac{1}{N}(\mu^2+\sigma^2) = \mu^2 + \frac{1}{N}\sigma^2 \end{split} \end{equation*}$$ Where I have used the fact that for $$n \neq m$$, $$x_n, x_m$$ are independent variables drawn from the same Gaussian distribution $$\mathcal{N}(x|\mu, \sigma^2)$$, along with the result that $$\mathbb{E}[x^2] = \mu^2 + \sigma^2$$ for $$x$$ drawn from $$\mathcal{N}(x|\mu, \sigma^2)$$.

$$\begin{equation*} \begin{split} \mathbb{E}[\sigma^2_{\text{ML}}] = \mathbb{E}\left[ \frac{1}{N}\sum_{n=1}^{N}(x_n - \mu_{\text{ML}})^2 \right] = \frac{1}{N}\sum_{n=1}^{N}\mathbb{E}[(x_n - \mu_{\text{ML}})^2] = \frac{1}{N}\sum_{n=1}^{N}\mathbb{E}[(x_n - \mu + \mu -\mu_{\text{ML}})^2] = \frac{1}{N}\sum_{n=1}^{N}\mathbb{E}[(x_n - \mu)^2 + (\mu -\mu_{\text{ML}})^2 + 2(x_n-\mu)(\mu - \mu_{\text{ML}})] \\= \mathbb{E}\left[\frac{1}{N}\sum_{n=1}^{N}(x_n - \mu)^2\right] + \mathbb{E}\left[\frac{1}{N}\sum_{n=1}^{N}(\mu -\mu_{\text{ML}})^2\right] + 2\mathbb{E}\left[\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu)(\mu - \mu_{\text{ML}})\right] \end{split} \end{equation*}$$

This can be evaluated term-by-term. The first term evaluates to

$$\mathbb{E}\left[\frac{1}{N}\sum_{n=1}^{N}(x_n - \mu)^2\right] = \mathbb{E}\left[\frac{1}{N}\sum_{n=1}^{N}(x_n^2 + \mu^2 - 2x_n\mu) \right] = \frac{1}{N}\sum_{n=1}^{N}(\mu^2 + \sigma^2 + \mu^2 - 2\mu^2) = \sigma^2$$

Note that $$\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu) = \mu_{\text{ML}} - \mu$$

Using this result, we can evaluate the second and third terms together $$\mathbb{E}\left[\frac{1}{N}\sum_{n=1}^{N}(\mu -\mu_{\text{ML}})^2\right] + 2\mathbb{E}\left[\frac{1}{N}\sum_{n=1}^{N}(x_n-\mu)(\mu - \mu_{\text{ML}})\right] = -\mathbb{E}[(\mu - \mu_{\text{ML}})^2]$$

$$\mathbb{E}[(\mu - \mu_{\text{ML}})^2] = \mathbb{E}[\mu^2 + \mu_{\text{ML}}^2 - 2\mu\mu_{\text{ML}}] = \frac{1}{N}\sigma^2$$

Combining the results, we get

$$\mathbb{E}[\sigma^2_{\text{ML}}] = \sigma^2 - \frac{1}{N}\sigma^2 = \frac{N-1}{N}\sigma^2$$

Quite a nice problem!