7
$\begingroup$

Consider estimating the variance of a RV $X$, we start with the sample variance:

$$ \begin{array}{ll} V_1 & = \frac{1}{N-1} \sum_{i=1}^N (X_i - \bar{X})^2\\ & = \frac{1}{N-1} \left(\sum_{i=1}^N X_i^2 - 2\bar{X}\sum_{i=1}^N X_i + N\bar{X}^2 \right)\\ &= \frac{1}{N-1} \left(\sum_{i=1}^N X_i^2 - N\bar{X}^2\right) \end{array} $$

Where $\bar{X} = N^{-1}\sum_{i=1}^N X_i$.

My question is, imagine we had a better estimate of the population mean: $\hat{X}$ that was guaranteed to have lower variance than the sample mean $\bar{X}$. Could it be used to obtain an improved estimate of the population variance? Also, notice how the above derivation no longer holds:

** Correction: previous question used $\frac{1}{N-1}$ which is only need if the sample mean is used. ** $$ \begin{array}{ll} V_2 & = \frac{1}{N} \sum_{i=1}^N (X_i - \hat{X})^2\\ & = \frac{1}{N} \left(\sum_{i=1}^N X_i^2 - 2\hat{X}\sum_{i=1}^N X_i + N\hat{X}^2 \right)\\ &\ne \frac{1}{N} \left(\sum_{i=1}^N X_i^2 - N\hat{X}^2\right) \end{array} $$

Unless $\bar{X} = \hat{X}$. Also, is there a way to measure the difference between $var(V_1)$ and $var(V_2)$ as a function of $var(\bar{X})$ and $var(\hat{X})$?

Edit: Let me further motivate why this might be useful. Imagine we have $K$ different estimators of the same mean, for simplicity imagine each of these are importance sampling estimators each with a different proposal distribution. So each of these estimators has a very different variance, and I want to know which estimator is the best one (or likely to be).

We could take $N$ samples from each estimator and estimate the sample variance for each one using the estimator $V_1$. Alternatively, we could use the samples from all of them together to get an improved estimate of the population mean and use $V_2$ instead. I would expect this would be advantageous if $K$ is very large (like $K = N^2$).

Consider a simple illustration, where one estimator has extremely high variance and returns $0$ in almost every single sample except with very low probability returns a huge value (enough to make it an unbiased estimate still). The sample variance ($V_1$) for this estimator is likely to be zero for any reasonable $N$, where $V_2$ would indicate that the estimator is actually quite poor.

$\endgroup$
  • $\begingroup$ Aren't $V_1$ and $V_2$ the same? $\endgroup$ – mark999 Apr 9 '12 at 6:16
  • $\begingroup$ What is "estimate of the sample mean"? Estimate can be about population mean only. $\endgroup$ – ttnphns Apr 9 '12 at 6:30
  • $\begingroup$ Thanks guys, I've edited my question, should be more clear. $\endgroup$ – fairidox Apr 9 '12 at 7:10
  • 4
    $\begingroup$ If you know the population mean and can use it instead of the sample mean, yes, it improves your estimate of variance, but the change to your estimate of the variance is just to use $N$ instead of $N-1$. A useful place to look to answer your question is the justification for using $N-1$ in the first place. $\endgroup$ – Peter Ellis Apr 9 '12 at 7:44
  • 2
    $\begingroup$ Lurking here in the background is a need for the definition of "best" in terms of the variance estimate. For example, in a Gaussian setting, neither $V_1$ nor the MLE $N V_1/(N-1)$ is best in the mean-squared sense. $\endgroup$ – cardinal Apr 9 '12 at 12:27
7
$\begingroup$

I don't think you're going to get a lot of benefit here, at least in situations where the mean is not very informative about the variance. In these situations, whether the $X_i$ vary about a number close to $\bar X$ or close to $\hat X$ is not very helpful in determining how much they vary.

To see this in the algebra, notice that $$ V_2 = \frac{1}{N-1}(\sum X_i - \hat X)^2 = \frac{1}{N-1}(\sum_i X_i - \bar X + \bar X - \hat X)^2 $$ meaning we can write it as $$ V_1 + \frac{N}{N-1}(\bar X - \hat X)^2. $$ Also, if you have $K$ different estimates of the same quantity, optimality is usually obtained by taking their weighted average, where the weights are proportional to the inverse of their (co)variance. The Gauss Markov Theorem is the main result, and its generalization by Aitken.

NB for a situation where the mean is very informative about the variance, use binary $X_i$, where if you know the mean you know the variance.

$\endgroup$
  • $\begingroup$ Awesome, I was able to verify this, well actually: $V_2 = \frac{N-1}{N}V_1 + (\bar{X} - \hat{X})^2$. When I use the edited/corrected version of $V_2$. $\endgroup$ – fairidox Apr 10 '12 at 7:02
0
$\begingroup$

First, $\bar X$ is the best linear unbiased estimator of $\mu$, the population mean. You can improve the mean squared error by shrinkage, but among unbiased estimates, you cannot get any better in terms of variance than $\sigma^2/n$ ($\sigma^2$ being the population variance). Moreover, from the asymptotic theory of estimating equations, I believe you cannot do better than $\sigma^2/n$ asymptotically, anyway, so the bias, if any, has to go away at a rate faster than $O(n^{-1/2})$. A technically valid source to improve efficiency of the estimator of the mean is knowledge of the distributional form of your data (gamma? Poisson? double exponential?), whereas the mean is expressed as a function of the (estimated) population parameters, but of course practicality of any such assumption is dubious, at best.

Second, the estimate $V_1$ is only good as an unbiased estimate of the variance. As far as I recall my stat theory classes, the estimate $$V_3=\frac1{n+1} \sum_i (X_i-\bar X)^2$$ has the smallest MSE as the estimator of the sampling variance $V[\bar X]$ (you probably have to assume normality of $X_i$'s to get a specific answer, as the MSE of the variance estimator depends on the kurtosis of the original distribution).

So you can do all sorts of things with biased estimates, and improve the MSE of either the estimator of $\mu$ or the estimator of $V[\hat\mu]$. But the unbiased estimation theory is fairly rigid, and Cramer-Rao bound together with Rao-Blackwell theorem give strict limits for your efficiency.

$\endgroup$
  • $\begingroup$ Thanks for the answer, although I'm interesting in estimating the population variance not the mean, and I have (I assume) a better approximation of the population mean (Xˆ) available to me, the question is how to use it and what the advantage may be. I'm not really looking to explore bias/variance tradeoff... I've edited my question a bit, perhaps the example makes my problem more clear. $\endgroup$ – fairidox Apr 10 '12 at 2:25
  • 1
    $\begingroup$ I see. Then my answer is quite a bit off indeed, I did not read your question carefully enough. Let me think if I can meaningfully edit my answer to address your question better. $\endgroup$ – StasK Apr 10 '12 at 3:12
-1
$\begingroup$

imagine we had a better estimate of the population mean: Xˆ that was guaranteed to have lower variance than the sample mean Xˉ. Could it be used to obtain an improved estimate of the population variance?

Yes.

Given only the samples the distribution over the mean and variance is normal-inverse-gamma. If you project this distribution so as to isolate only the estimate of the variance, it's no longer inverse-gamma-distributed, but will have a heavier tail.

On the other hand, given the samples and an exact mean, the distribution over the variance is inverse-gamma with shape $\frac n2$.

How you turn these priors over the variance into an estimate is up to you. The maximum likelihood estimators that most people calculate are the modes of the aforementioned distributions. I don't think the two distributions will have the same mode. (I think the first one will usually be larger, but the second will be larger if your true mean is very far from the sample mean.)

$\endgroup$
  • $\begingroup$ I have a problem with a first statement. Do you imply that for any given sample, the joint distribution of sample mean and sample variance is normal-inverse-gamma? $\endgroup$ – mpiktas Apr 10 '12 at 6:45
  • $\begingroup$ @mpiktas: I meant the distribution over the true mean and variance. It's true that I'm making an assumption that the data was generated by a normal process, but that's not a big assumption since all of the variance estimation techniques assume something similar. (If we had additional information about the generating distribution, any variance estimation method would have to take it into account.) $\endgroup$ – Neil G Apr 10 '12 at 6:50
  • $\begingroup$ Concerning your flag, Neil: I cannot find any evidence to support your suspicions. We'll keep an eye out for any voting irregularities, of course, but I don't think there's a problem here. $\endgroup$ – whuber Sep 23 '12 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.