Why are the MLE and MMSE corrections for sample variances different?

Question

I have a number of samples of sample size 2 and a number of sample of sample size 3. If my samples are all samples from populations with a shared population variances, I wish to estimate population variance. I wish to estimate population variance in the way that will give me a value as close as theoretically possible (with the information I possess) to the true value of population variance.

While struggling with the question of how the sample numbers (planning to average corrected values) affect the sample variance corrections I should be using (because increased sample number of averaged sample values decreases the average's variance without decreasing its bias), I came across another concept I was unfamiliar with.

In the accepted answer to Maximum Likelihood Estimation — why it is used despite being biased in many cases, it is said that the corrected denominator for an unbiased estimator is $n-1$ (which I knew), that the corrected denominator for the minimum mean-squared error estimator is $n+1$ (which I knew), and that the corrected denominator for the maximum likelihood estimator is $n$ (which I did not know).

If I understand correctly, each of these is corrected denominator for when estimating a population variance from a single sample (rather than an average of values from multiple samples), thus I cannot use them directly in any case.

Reiterating, I wish to use the approach which will give me the value theoretically closest to the true population variance value, given the information available to me. I thought this would be the approach which gives the minimum mean squared error ('getting arrows clustered as closely to the bullseye as possible'), but then how can the maximum likelihood estimator (the value which is most likely to give rise to the observed data) be different?

To rephrase, given that I want to get a final value which is close as possible to the true value (of the population variance), which of these approaches--MMSE and MLE--is the wrong approach, and why?

(Of course, if there is yet another approach which serves my intention even better that I haven't heard about yet I would be glad to learn of it!)

Answer 1

how can the maximum likelihood estimator (the value which is most likely to give rise to the observed data) be different?

Describing the MLE as "the value which is most likely to give rise to the observed data" is at best very misleading.

It is in fact the value that makes the data more likely than they would be with any other value.

Suppose you want $\widehat\sigma^{\,2}$ to be that function of the data (if one exists) that, for every value of $\sigma^2,$ gives you the smallest mean squared error of estimation of $\sigma^2,$ i.e. you're minimizing $\operatorname E\left( \left( \widehat\sigma^{\,2} - \sigma^2 \right) ^2 \right).$

The result is not the same as if you're minimizing $\operatorname E\left( \left( \widehat\sigma - \sigma \right)^2 \right)$ or $\operatorname E\left( h \left( \left|\widehat\sigma^{\,3} - \sigma^3\right| \right) \right)$ or some other function of $\widehat\sigma^{\,2}$ and $\sigma^2.$

But the maximum-likelihood distribution is the same regardless of which function of $\sigma^2$ is used in parameterizing the family of distributions. You have a family of probability distributions, in this case all normal distributions, and the method of maximum likelihood singles out one of them as the estimate. It's the same one regardless of which of many parameterizations is used.

Answer 2

Comment continued:

Here is a brief simulation with standard normal data, for which $\sigma^2 = 1.$ In the simulation v1 is a vector a million unbiased estimates $S^2,$ each for a sample of size $n = 10.$

set.seed(1116)
m = 10^6; n = 10; 
x = rnorm(m*n); DTA = matrix(x, nrow=m)

v1 = apply(DTA, 1, var)  # denom n-1
mean(v1);  mean((v1-1)^2)
[1] 0.9998608  # aprx 1
[1] 0.2222156  # aprx MSE of unbiased variance estimator

v2 = (n-1)*v1/(n+1)      # denom n+1
mean(v2);  mean((v2-1)^2)
[1] 0.8180679  # noticeable bias 
[1] 0.1818552  # smaller MSE

Formal analytic proofs are not difficult.

Addendum per Comment. Although for normal data the usual definition of the sample variance (denominator $n-1$ gives unbiased estimates of population variance $\sigma^2,$ unbiasedness does not survive the nonlinear operation of taking square roots. Thus it is not true that $E(S) = \sigma.$ In particular, for normal samples of size $n=3,$ one has $E(S) \approx 0.866\sigma,$ so that $E(S/0.866) = E(1.155\,S) = \sigma,$

set.seed(1118)
s = replicate(10^6, sd(rnorm(3)))
mean(s)
[1] 0.8863831
k = 1/mean(s)
mean(k*s)
[1] 1

Answer 3

An example that is much more illustrative would be the maximum likelihood estimate of a uniform distribution $U(0,\theta)$ whose estimate will be the maximum of a sample and it follows a scaled beta distribution (a Pearson type I distribution).

$$f\left(\hat\theta\right) = \begin{cases} \frac{n}{\theta^n} \hat{\theta}^{n-1} &\qquad \text{for $0 < \hat\theta \leq \theta$} \\ 0 & \qquad \text{else} \end{cases}$$

This could look like

We always have negative error $\hat{\theta} < \theta$ and it is not difficult to imagine that some scaling will improve the squared error because the amount by which the negative errors decrease will be much larger than the amount by which the positive errors increase.

---

The estimate of the mean of a Gaussian distribution is another example to visualise this idea. In this case we use shrinkage and, the amount by which the positive errors decrease will be much larger than the amount by which the negative errors increase. This is because, if you scale a variable, then this will have a larger effect on larger values.

Below is a simulation for 100 estimates of sample means, when the true mean is equal to 1. The MLE will be distributed unbiased around 1. When we apply a shrinkage, then the vakues above 1 (in red) will shift more than the values below 1 (in blue). The bias of shrinking will make some errors larger and others smaller. The net change can be an improvement.

In the image below arrows show which place the estimates would become when we shrink with a factor 0.97, the arrows on the right (improvements) are larger than the arrows on the left (detoriations).