Unbiased estimator for variance or Maximum Likelihood Estimator?

Question

I know that $\frac{1}{n-1}\sum_{i=1}^{n}(X-\bar{X_{n}})^{2}$ is an unbiased estimator for the variance.

I thought of such an estimator to be useful when it is not know from which distribution the data at hand are coming from.

Now given I have a data set and I do know which is the underlying distributions of the observations. Shouldn't I prefer a Maximum Likelihood Estimator for the variance even if it is biased since it obviously makes use of the underlying distribution of the data over the general variance estimator cited above, which is not for any specific distribution?

Thanks

Answer 1

I think the answer is generally yes. If you know more about a distribution then you should use that information. For some distributions this will make very little difference, but for other it could be considerable.

As an example, consider the poisson distribution. In this case the mean and the variance are both equal to the parameter $\lambda$ and the ML estimate of $\lambda$ is the sample mean.

The charts below show 100 simulations of estimating the variance by taking the mean or the sample variance. The histogram labelled X1 is the using sample mean, and X2 is using the sample variance. As you can see, both are unbiased but the mean is a much better estimate of $\lambda$ and hence a better estimate of he variance.

The R code for the above is here:

library(ggplot2)
library(reshape2)
testpois = function(){
  X = rpois(100, 4)
  mu = mean(X)
  v = var(X)
  return(c(mu, v))
}

P = data.frame(t(replicate(100, testpois())))
P = melt(P)

ggplot(P, aes(x=value)) + geom_histogram(binwidth=.1, colour="black", fill="white") +
  geom_vline(aes(xintercept=mean(value, na.rm=T)),   # Ignore NA values for mean
             color="red", linetype="dashed", size=1) + facet_grid(variable~.)

As to the question of bias, I wouldn't worry too much about your estimator being biased (in the example above it isn't, but that is just luck). If unbiasedness is important to you you can always use Jackknife to try remove the bias.

Answer 2

I have moved my comment to an answer so I can expand on it as requested.

[If you mean the variance form $\frac{1}{n}\sum_{i=1}^{n}(X-\bar{X_{n}})^{2}$ as ML (which it is for the normal), then both forms use exactly the same information - the sums of squares of deviation from the mean. The only difference is that scaling factor.]

If you need the variance estimate to be unbiased you could use it (note that in general you could take any MLE for the variance at a particular distribution and see if you can at least approximately unbias that; it may be more efficient), but it's not (say) minimum MSE for the variance, and it's not unbiased if you're taking the square root and using that for the standard deviation.

At least the ML estimate for the variance is still ML for the s.d. (irrespective of which distribution for you have an MLE of the variance).

Here's why I say that:

MLE's have the property of being invariant to transformation of parameters - the MLE of $g(\theta)$ is $g(\hat{\theta})$ (or more concisely, $\widehat{g(\theta)}=g(\hat{\theta})$). See the brief discussion here, and the stuff under note 2 here.

None of those prove it, but I'll give you a (somewhat handwavy) motivation/outline of an argument for the simple case of monotonic transformations. You can find a complete argument in many texts that discuss ML at more than a really elementary level.

In the case of monotonic transformations: Take a simple case - imagine I have some curve ($y$ vs $x$) with a single peak somewhere in the middle (both a global and local maximum). Now I transform the $x$ to $\xi$ ($\xi=t(x)$) while $y$ is unchanged. The shape of the curve changes, but the corresponding $y$'s don't. The original maximum of $y$ is still the same maximum at the corresponding place in $\xi$ as it was under $x$ (that is, if the maximum was at $x^*$, it's now at $\xi^*=t(x^*)$. You should see how to extend that intuition to a monotonic transformation and any global maximum. [The more general case of non-monotonic transformations is less immediately obvious, but is still true. Edit: It's true in the case of one-to-one functions by a similar argument to the above.]

Returning to the original answer:

In practice (in the the $n$ vs $n-1$ case) there's rarely much difference and I regularly use each in different circumstances with little worry. I'm usually not worried about an unbiased variance estimate