7
$\begingroup$

I know that $\frac{1}{n-1}\sum_{i=1}^{n}(X-\bar{X_{n}})^{2}$ is an unbiased estimator for the variance.

I thought of such an estimator to be useful when it is not know from which distribution the data at hand are coming from.

Now given I have a data set and I do know which is the underlying distributions of the observations. Shouldn't I prefer a Maximum Likelihood Estimator for the variance even if it is biased since it obviously makes use of the underlying distribution of the data over the general variance estimator cited above, which is not for any specific distribution?

Thanks

$\endgroup$
  • 1
    $\begingroup$ Note that the ML estimator has just a $\frac{1}{n}$ factor instead of $\frac{1}{n-1}$. For moderately large $n$'s, using one or the other will probably make no difference at all. $\endgroup$ – ocram Feb 8 '13 at 21:51
  • 5
    $\begingroup$ @ocram That's the ML estimator for a Normal distribution (and, possibly, a few others), but it's not universal. For instance, no multiple of a sum of squares of deviations from the mean is an ML estimator of the variance of a Poisson distribution. $\endgroup$ – whuber Feb 8 '13 at 22:50
  • 1
    $\begingroup$ Both forms use exactly the same information - the sums of squares of deviation from the mean. The only difference is that scaling factor. If you need the variance estimate to be unbiased you should use it, but it's not (say) minimum MSE for the variance, and it's not unbiased if you're taking the square root and using that for the standard deviation. At least the ML estimate (ML at the normal) is still ML for the s.d. In practice there's rarely much difference and I regularly use each in different circumstances with little worry. I'm usually not worried about an unbiased variance estimate. $\endgroup$ – Glen_b Feb 8 '13 at 22:53
  • $\begingroup$ Thanks for the answers. @Glen_b: Could you possibly elaborate on the statement that the ML estimate is ML for the s.d.? Best, Pegah $\endgroup$ – Pegah Feb 8 '13 at 23:07
  • 1
    $\begingroup$ See also the last paragraph of Glen_b's answer in the thread "Maximum Likelihood Estimation — why it is used despite being biased in many cases". $\endgroup$ – Richard Hardy Apr 26 '18 at 9:00
9
$\begingroup$

I think the answer is generally yes. If you know more about a distribution then you should use that information. For some distributions this will make very little difference, but for other it could be considerable.

As an example, consider the poisson distribution. In this case the mean and the variance are both equal to the parameter $\lambda$ and the ML estimate of $\lambda$ is the sample mean.

The charts below show 100 simulations of estimating the variance by taking the mean or the sample variance. The histogram labelled X1 is the using sample mean, and X2 is using the sample variance. As you can see, both are unbiased but the mean is a much better estimate of $\lambda$ and hence a better estimate of he variance.

enter image description here

The R code for the above is here:

library(ggplot2)
library(reshape2)
testpois = function(){
  X = rpois(100, 4)
  mu = mean(X)
  v = var(X)
  return(c(mu, v))
}

P = data.frame(t(replicate(100, testpois())))
P = melt(P)

ggplot(P, aes(x=value)) + geom_histogram(binwidth=.1, colour="black", fill="white") +
  geom_vline(aes(xintercept=mean(value, na.rm=T)),   # Ignore NA values for mean
             color="red", linetype="dashed", size=1) + facet_grid(variable~.)

As to the question of bias, I wouldn't worry too much about your estimator being biased (in the example above it isn't, but that is just luck). If unbiasedness is important to you you can always use Jackknife to try remove the bias.

$\endgroup$
  • 1
    $\begingroup$ That's a very good answer, thank you a lot. Indeed I was not mainly interested in bias, but thought of bias as an example of a criterion which could be used to prefer one approach over the other. So, I was generally interested in knowing if there are theoretical reasons to rather prefer the ML estimate, since as you say intuitively it makes more sense to use all the information available. $\endgroup$ – Pegah Feb 9 '13 at 14:46
2
$\begingroup$

I have moved my comment to an answer so I can expand on it as requested.

[If you mean the variance form $\frac{1}{n}\sum_{i=1}^{n}(X-\bar{X_{n}})^{2}$ as ML (which it is for the normal), then both forms use exactly the same information - the sums of squares of deviation from the mean. The only difference is that scaling factor.]

If you need the variance estimate to be unbiased you could use it (note that in general you could take any MLE for the variance at a particular distribution and see if you can at least approximately unbias that; it may be more efficient), but it's not (say) minimum MSE for the variance, and it's not unbiased if you're taking the square root and using that for the standard deviation.

At least the ML estimate for the variance is still ML for the s.d. (irrespective of which distribution for you have an MLE of the variance).

Here's why I say that:

MLE's have the property of being invariant to transformation of parameters - the MLE of $g(\theta)$ is $g(\hat{\theta})$ (or more concisely, $\widehat{g(\theta)}=g(\hat{\theta})$). See the brief discussion here, and the stuff under note 2 here.

None of those prove it, but I'll give you a (somewhat handwavy) motivation/outline of an argument for the simple case of monotonic transformations. You can find a complete argument in many texts that discuss ML at more than a really elementary level.

In the case of monotonic transformations: Take a simple case - imagine I have some curve ($y$ vs $x$) with a single peak somewhere in the middle (both a global and local maximum). Now I transform the $x$ to $\xi$ ($\xi=t(x)$) while $y$ is unchanged. The shape of the curve changes, but the corresponding $y$'s don't. The original maximum of $y$ is still the same maximum at the corresponding place in $\xi$ as it was under $x$ (that is, if the maximum was at $x^*$, it's now at $\xi^*=t(x^*)$. You should see how to extend that intuition to a monotonic transformation and any global maximum. [The more general case of non-monotonic transformations is less immediately obvious, but is still true. Edit: It's true in the case of one-to-one functions by a similar argument to the above.]

Returning to the original answer:

In practice (in the the $n$ vs $n-1$ case) there's rarely much difference and I regularly use each in different circumstances with little worry. I'm usually not worried about an unbiased variance estimate

$\endgroup$
  • 2
    $\begingroup$ (-1) nowhere in the question does it state that his data is normal. OP is asking, if knows the underlying family of distributions (Poisson, Normal, Gamma, or whatever) should he base his estimator on ML even if it is biased, or just use the generic (nonparametric) unbiased estimator anyways. For the Poisson distribution, there is a lot more than just a scaling factor separating the MLE of the variance and the unbiased estimator the OP gave. $\endgroup$ – guy Feb 8 '13 at 23:40
  • $\begingroup$ When the original question calls the $\frac{1}{n}$ form of variance "ML", that implies normality. That is, I introduced no assumption not made by the question. I merely pointed that assumption out. You're downvoting my answer because I mention that implied assumption explicitly rather than have it merely implied? What? $\endgroup$ – Glen_b Feb 8 '13 at 23:44
  • 2
    $\begingroup$ Nowhere in the OP do I see that scaling factor, and as far as I can tell it hasn't been edited... $\endgroup$ – guy Feb 8 '13 at 23:45
  • $\begingroup$ @guy Okay, that's true. I was conflating ocram's comment (which the OP didn't contradict as not being applicable) with the question. I have made some edits. $\endgroup$ – Glen_b Feb 8 '13 at 23:52
  • 2
    $\begingroup$ Although I do understand the invariance property of ML, I am not sure that the original question has been fully adressed (and as @guy has correctly pointed out I did not assume any specific family of distributions). Pegah $\endgroup$ – Pegah Feb 9 '13 at 0:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.