Will MLE always produce Biased Variance Estimates?

I am trying to understand the following points:

• Does MLE (Maximum Likelihood Estimate) always produce biased variance estimates?
• Does RMLE (Restricted Maximum Likelihood Estimate) always produce unbiased variance estimates?

Part 1: As an example, I tried to show that in the case of a Normal Distribution, the MLE for the variance is biased:

For a sample data as $$x_1, x_2, ..., x_n$$, the parameters of the normal distribution as $$\mu$$ (mean) and $$\sigma^2$$ (variance). The likelihood function is:

$$L(\mu, \sigma^2 | x_1, ..., x_n) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x_i-\mu)^2}{2\sigma^2}}$$

$$l(\mu, \sigma^2 | x_1, ..., x_n) = -\frac{n}{2} \log(2\pi) - \frac{n}{2} \log(\sigma^2) - \frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i - \mu)^2$$

Taking the derivative with respect to $$\mu$$ and setting it equal to zero gives:

$$\frac{\partial l}{\partial \mu} = \frac{1}{\sigma^2} \sum_{i=1}^{n} (x_i - \mu) = 0$$

$$\hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} x_i = \bar{x}$$

Taking the derivative with respect to $$\sigma^2$$ and setting it equal to zero gives:

$$\frac{\partial l}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2(\sigma^2)^2} \sum_{i=1}^{n} (x_i - \mu)^2 = 0$$

$$\hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^{n} (x_i - \bar{x})^2$$

However, we can see that this will be a biased estimate:

$$E[\hat{\sigma}^2] = E\left[\frac{1}{n} \sum_{i=1}^{n} (x_i - \bar{x})^2\right]$$ $$E[\hat{\sigma}^2] = \frac{1}{n} \sum_{i=1}^{n} E[x_i^2 - 2x_i\bar{x} + \bar{x}^2]$$ $$E[\hat{\sigma}^2] = \sigma^2 - \frac{\sigma^2}{n}$$

Based on my naive understanding of this, I think that this result (i.e. biased variance estimate) is inevitable when using MLE? The reason I was told was the following: the MLE variance estimate itself depends on the MLE mean estimate. Therefore, this dependence somehow creates a bias. However, this is not clear to me - if an estimate depends on another estimate, why does this always result in the MLE being biased? Can this mathematically be proven? If an estimate depends on another estimate - will it necessarily be biased when estimated via MLE?

Part 2:

In RMLE, we prevent the variance estimate from depending on the mean estimate by removing the mean estimate from the likelihood (e.g. via Gauss-Hermite integration):

$$L_R(\sigma^2 | x_1, ..., x_n) = \int L(\mu, \sigma^2 | x_1, ..., x_n) d\mu$$ $$L_R(\sigma^2 | x_1, ..., x_n) \approx \sum_{i=1}^{n} w_i L(\mu_i, \sigma^2 | x_1, ..., x_n)$$

From here, we proceed and use standard MLE on this Restricted Likelihood $$L_R$$ and solve for $$\sigma$$. Since we have removed the mean parameter, the variance estimate will not depend on the mean. Therefore, it will not be biased. However, it is difficult to see/accept this in situations where there is no closed form solution. (I am aware that in this particular example with a Normal Distribution, the RMLE has a closed form solution and it can be verified that the estimate is unbiased, but this is not usually possible)

But is there a mathematical proof for any of this?

• Does MLE (Maximum Likelihood Estimate) always produce biased variance estimates?
• Does RMLE (Restricted Maximum Likelihood Estimate) always produce unbiased variance estimates?
• If an estimate depends on another estimate - will it necessarily be biased when estimated via MLE?
• i made some mistakes... i am correcting them Feb 24 at 18:12

2. The logic "the variance estimate will not depend on [the estimate of] the mean... Therefore it will not be biased" is fallacious. (Note what I've added in brackets!) Consider a Negative Binomial distribution with a known mean $$\mu$$. The variance equals $$\mu/p$$, where $$p$$ is the probability parameter. In this case, the MLE of $$p$$ does not provide an unbiased estimate of $$1/p$$, which would be required for an unbiased estimator of the variance.
3. No; consider your Normal distribution example. One can write the unbiased estimate of $$\sigma^2$$ as $$n/(n-1) \cdot \hat{\sigma}^2$$. Therefore, we have constructed another estimator of the variance based on the MLE of the variance, and it is unbiased.