Prove that maximum likelihood estimators for Gaussian distribution are a global maximum

Question

I'm studying on Casella-Berger, I'm at page 322 in which it explain how to find MLE for a Gaussian distribution with parameter $\mu$ and $\sigma^2$, both unknown. It finds MLE, and up to this point it is all clear, and they are $\hat{\mu} = \bar{x}$ and $\hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^{n} (x_i-\bar{x})^2$.

Now, it says it is difficult to prove analitically that these are global maxima indeed, and it uses this fact:

If $\theta \ne \bar{x}$ then $\sum (x_i-\theta)^2 > \sum (x_i-\bar{x})^2$.

It doesn't give any explaination for that. Is there something obvious I don't see?

Answer 1

You can use a sum-of-squares argument to see this.

$$\sum_i (x_i-\theta)^2 = \sum_i (x_i - \bar{x}+\bar{x}-\theta)^2 = \sum_i (x_i-\bar{x})^2+n(\bar{x}-\theta)^2+\\\color{red}{2(\bar{x}-\theta)\sum_i(x_i-\bar{x})}$$

Now, $\bar{x}$ is defined so that the cross term becomes zero since $\sum_i x_i = \sum_i \bar{x}$.

We are therefore left with:

$$\sum_i (x_i - \theta)^2 = \sum_i (x_i-\bar{x})^2+n(\bar{x}-\theta)^2$$

This is the variance/bias decomposition of the squared-error associated with estimator $\theta$. Since both terms on the RHS are positive and only the bias term depends on $\theta$, we have:

$$\sum_i (x_i - \theta)^2 > \sum_i (x_i - \bar{x})^2$$

Answer 2

$\theta\mapsto \sum (x_i-\theta)^2$ is a quadratic function in $\theta$ that opens upwards. It has a unique minimum where its derivative is $0$, that is to say when $2\sum (x_i-\theta)=0$ ie $\theta = \hat x$.