A short question on sample variance

Question

Consider estimating the variance of a RV $X$, we start with the sample variance:

$$ \begin{array}{ll} V_1 & = \frac{1}{N-1} \sum_{i=1}^N (X_i - \bar{X})^2\\ & = \frac{1}{N-1} \left(\sum_{i=1}^N X_i^2 - 2\bar{X}\sum_{i=1}^N X_i + N\bar{X}^2 \right)\\ &= \frac{1}{N-1} \left(\sum_{i=1}^N X_i^2 - N\bar{X}^2\right) \end{array} $$

Where $\bar{X} = N^{-1}\sum_{i=1}^N X_i$.

My question is, imagine we had a better estimate of the population mean: $\hat{X}$ that was guaranteed to have lower variance than the sample mean $\bar{X}$. Could it be used to obtain an improved estimate of the population variance? Also, notice how the above derivation no longer holds:

** Correction: previous question used $\frac{1}{N-1}$ which is only need if the sample mean is used. ** $$ \begin{array}{ll} V_2 & = \frac{1}{N} \sum_{i=1}^N (X_i - \hat{X})^2\\ & = \frac{1}{N} \left(\sum_{i=1}^N X_i^2 - 2\hat{X}\sum_{i=1}^N X_i + N\hat{X}^2 \right)\\ &\ne \frac{1}{N} \left(\sum_{i=1}^N X_i^2 - N\hat{X}^2\right) \end{array} $$

Unless $\bar{X} = \hat{X}$. Also, is there a way to measure the difference between $var(V_1)$ and $var(V_2)$ as a function of $var(\bar{X})$ and $var(\hat{X})$?

Edit: Let me further motivate why this might be useful. Imagine we have $K$ different estimators of the same mean, for simplicity imagine each of these are importance sampling estimators each with a different proposal distribution. So each of these estimators has a very different variance, and I want to know which estimator is the best one (or likely to be).

We could take $N$ samples from each estimator and estimate the sample variance for each one using the estimator $V_1$. Alternatively, we could use the samples from all of them together to get an improved estimate of the population mean and use $V_2$ instead. I would expect this would be advantageous if $K$ is very large (like $K = N^2$).

Consider a simple illustration, where one estimator has extremely high variance and returns $0$ in almost every single sample except with very low probability returns a huge value (enough to make it an unbiased estimate still). The sample variance ($V_1$) for this estimator is likely to be zero for any reasonable $N$, where $V_2$ would indicate that the estimator is actually quite poor.

Answer 1

I don't think you're going to get a lot of benefit here, at least in situations where the mean is not very informative about the variance. In these situations, whether the $X_i$ vary about a number close to $\bar X$ or close to $\hat X$ is not very helpful in determining how much they vary.

To see this in the algebra, notice that $$ V_2 = \frac{1}{N-1}(\sum X_i - \hat X)^2 = \frac{1}{N-1}(\sum_i X_i - \bar X + \bar X - \hat X)^2 $$ meaning we can write it as $$ V_1 + \frac{N}{N-1}(\bar X - \hat X)^2. $$ Also, if you have $K$ different estimates of the same quantity, optimality is usually obtained by taking their weighted average, where the weights are proportional to the inverse of their (co)variance. The Gauss Markov Theorem is the main result, and its generalization by Aitken.

NB for a situation where the mean is very informative about the variance, use binary $X_i$, where if you know the mean you know the variance.

Answer 2

First, $\bar X$ is the best linear unbiased estimator of $\mu$, the population mean. You can improve the mean squared error by shrinkage, but among unbiased estimates, you cannot get any better in terms of variance than $\sigma^2/n$ ($\sigma^2$ being the population variance). Moreover, from the asymptotic theory of estimating equations, I believe you cannot do better than $\sigma^2/n$ asymptotically, anyway, so the bias, if any, has to go away at a rate faster than $O(n^{-1/2})$. A technically valid source to improve efficiency of the estimator of the mean is knowledge of the distributional form of your data (gamma? Poisson? double exponential?), whereas the mean is expressed as a function of the (estimated) population parameters, but of course practicality of any such assumption is dubious, at best.

Second, the estimate $V_1$ is only good as an unbiased estimate of the variance. As far as I recall my stat theory classes, the estimate $$V_3=\frac1{n+1} \sum_i (X_i-\bar X)^2$$ has the smallest MSE as the estimator of the sampling variance $V[\bar X]$ (you probably have to assume normality of $X_i$'s to get a specific answer, as the MSE of the variance estimator depends on the kurtosis of the original distribution).

So you can do all sorts of things with biased estimates, and improve the MSE of either the estimator of $\mu$ or the estimator of $V[\hat\mu]$. But the unbiased estimation theory is fairly rigid, and Cramer-Rao bound together with Rao-Blackwell theorem give strict limits for your efficiency.

Answer 3

imagine we had a better estimate of the population mean: Xˆ that was guaranteed to have lower variance than the sample mean Xˉ. Could it be used to obtain an improved estimate of the population variance?

Yes.

Given only the samples the distribution over the mean and variance is normal-inverse-gamma. If you project this distribution so as to isolate only the estimate of the variance, it's no longer inverse-gamma-distributed, but will have a heavier tail.

On the other hand, given the samples and an exact mean, the distribution over the variance is inverse-gamma with shape $\frac n2$.

How you turn these priors over the variance into an estimate is up to you. The maximum likelihood estimators that most people calculate are the modes of the aforementioned distributions. I don't think the two distributions will have the same mode. (I think the first one will usually be larger, but the second will be larger if your true mean is very far from the sample mean.)