9
$\begingroup$

I am trying to understand the following points:

  • Does MLE (Maximum Likelihood Estimate) always produce biased variance estimates?
  • Does RMLE (Restricted Maximum Likelihood Estimate) always produce unbiased variance estimates?

Part 1: As an example, I tried to show that in the case of a Normal Distribution, the MLE for the variance is biased:

For a sample data as $x_1, x_2, ..., x_n$, the parameters of the normal distribution as $\mu$ (mean) and $\sigma^2$ (variance). The likelihood function is:

$$L(\mu, \sigma^2 | x_1, ..., x_n) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x_i-\mu)^2}{2\sigma^2}}$$

$$l(\mu, \sigma^2 | x_1, ..., x_n) = -\frac{n}{2} \log(2\pi) - \frac{n}{2} \log(\sigma^2) - \frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i - \mu)^2$$

Taking the derivative with respect to $\mu$ and setting it equal to zero gives:

$$\frac{\partial l}{\partial \mu} = \frac{1}{\sigma^2} \sum_{i=1}^{n} (x_i - \mu) = 0$$

$$\hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} x_i = \bar{x}$$

Taking the derivative with respect to $\sigma^2$ and setting it equal to zero gives:

$$\frac{\partial l}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2(\sigma^2)^2} \sum_{i=1}^{n} (x_i - \mu)^2 = 0$$

$$\hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^{n} (x_i - \bar{x})^2$$

However, we can see that this will be a biased estimate:

$$E[\hat{\sigma}^2] = E\left[\frac{1}{n} \sum_{i=1}^{n} (x_i - \bar{x})^2\right]$$ $$E[\hat{\sigma}^2] = \frac{1}{n} \sum_{i=1}^{n} E[x_i^2 - 2x_i\bar{x} + \bar{x}^2]$$ $$E[\hat{\sigma}^2] = \sigma^2 - \frac{\sigma^2}{n}$$

Based on my naive understanding of this, I think that this result (i.e. biased variance estimate) is inevitable when using MLE? The reason I was told was the following: the MLE variance estimate itself depends on the MLE mean estimate. Therefore, this dependence somehow creates a bias. However, this is not clear to me - if an estimate depends on another estimate, why does this always result in the MLE being biased? Can this mathematically be proven? If an estimate depends on another estimate - will it necessarily be biased when estimated via MLE?

Part 2:

In RMLE, we prevent the variance estimate from depending on the mean estimate by removing the mean estimate from the likelihood (e.g. via Gauss-Hermite integration):

$$L_R(\sigma^2 | x_1, ..., x_n) = \int L(\mu, \sigma^2 | x_1, ..., x_n) d\mu$$ $$ L_R(\sigma^2 | x_1, ..., x_n) \approx \sum_{i=1}^{n} w_i L(\mu_i, \sigma^2 | x_1, ..., x_n) $$

From here, we proceed and use standard MLE on this Restricted Likelihood $L_R$ and solve for $\sigma$. Since we have removed the mean parameter, the variance estimate will not depend on the mean. Therefore, it will not be biased. However, it is difficult to see/accept this in situations where there is no closed form solution. (I am aware that in this particular example with a Normal Distribution, the RMLE has a closed form solution and it can be verified that the estimate is unbiased, but this is not usually possible)

But is there a mathematical proof for any of this?

  • Does MLE (Maximum Likelihood Estimate) always produce biased variance estimates?
  • Does RMLE (Restricted Maximum Likelihood Estimate) always produce unbiased variance estimates?
  • If an estimate depends on another estimate - will it necessarily be biased when estimated via MLE?
$\endgroup$
1
  • $\begingroup$ i made some mistakes... i am correcting them $\endgroup$ Feb 24 at 18:12

1 Answer 1

8
$\begingroup$
  1. No, MLE does not always produce biased variance estimates. Consider the MLE of the variance of the Poisson distribution, which is equal to the mean. The MLE is the sample mean, which is, of course, unbiased.

  2. The logic "the variance estimate will not depend on [the estimate of] the mean... Therefore it will not be biased" is fallacious. (Note what I've added in brackets!) Consider a Negative Binomial distribution with a known mean $\mu$. The variance equals $\mu/p$, where $p$ is the probability parameter. In this case, the MLE of $p$ does not provide an unbiased estimate of $1/p$, which would be required for an unbiased estimator of the variance.

  3. No; consider your Normal distribution example. One can write the unbiased estimate of $\sigma^2$ as $n/(n-1) \cdot \hat{\sigma}^2$. Therefore, we have constructed another estimator of the variance based on the MLE of the variance, and it is unbiased.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.