# Bias correction in weighted variance

For unweighted variance $$\text{Var}(X):=\frac{1}{n}\sum_i(x_i - \mu)^2$$ there exists the bias corrected sample variance, when the mean was estimated from the same data: $$\text{Var}(X):=\frac{1}{n-1}\sum_i(x_i - E[X])^2$$

I'm looking into weighted mean and variance, and wondering what the appropriate bias correction for the weighted variance is. Using: $$\text{mean}(X):=\frac{1}{\sum_i \omega_i}\sum_i \omega_i x_i$$

The "naive", non-corrected variance I'm using is this: $$\text{Var}(X):=\frac{1}{\sum_i \omega_i}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

So I'm wondering whether the correct way of correcting bias is

A) $$\text{Var}(X):=\frac{1}{\sum_i \omega_i - 1}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

or B) $$\text{Var}(X):=\frac{n}{n-1}\frac{1}{\sum_i \omega_i}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

or C) $$\text{Var}(X):=\frac{\sum_i \omega_i}{(\sum_i \omega_i)^2-\sum_i \omega_i^ 2}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

A) does not make sense to me when the weights are small. The normalization value could be 0 or even negative. But how about B) ($n$ is the number of observations) - is this the correct approach? Do you have some reference that shows this? I belive "Updating mean and variance estimates: an improved method", D.H.D. West, 1979 uses this. The third, C) is my interpretation of the answer to this question: http://mathoverflow.net/questions/22203/unbiased-estimate-of-the-variance-of-an-unnormalised-weighted-mean

For C) I have just realized that the denominator looks a lot like $\text{Var}(\Omega)$. Is there some general connection here? I think it does not entirely align; and obviously there is the connection that we are trying to compute the variance...

All three of them seem to "survive" the sanity check of setting all $\omega_i=1$. So which one should I used, under which premises? ''Update:'' whuber suggested to also do the sanity check with $\omega_1=\omega_2=.5$ and all remaining $\omega_i=\epsilon$ tiny. This seems to rule out A and B.

-
 When you consider cases where the two largest weights are equal and all the rest become vanishingly small, both (A) and (B) drop from contention (because they disagree with the known results for $n=2$). (C) appears to be an approximation; I suspect the correct factor is a much more complicated function of the weights. – whuber♦ Jan 9 at 15:44 @whuber ThePawn below suggests that it is C. Do you have more detailed concerns? – Anony-Mousse Jan 12 at 10:35

I went through the math and ended up with variant C:

$$Var(X) = \frac{(\sum_i \omega_i)^2}{(\sum_i \omega_i)^2 - \sum_i \omega_i^2}\overline V$$ where $\overline V$ is the non corrected variance estimation. The formula agrees with the unweighted case when all $\omega_i$ are identical. I detail the proof below:

Setting $\lambda_i = \frac{\omega_i}{\sum_i \omega_i}$, we have

$$\overline V = \sum_i \lambda_i (x_i - \sum_j \lambda_j x_j)^2$$

Expanding the inner term gives: $$(x_i - \sum_j \lambda_j x_j)^2 = x_i^2 + \sum_{j, k} \lambda_j \lambda_k x_j x_k - 2 \sum_j \lambda_j x_i x_j$$

If we take the expectation, we have that $E[x_i x_j] = Var(X)1_{i = j} + E[X]^2$, the term $E[X]$ being present in each term, it cancels out and we get:

$$E[\overline V] = Var(X) \sum_i \lambda_i (1 + \sum_j \lambda_j^2- 2 \lambda_i )$$ that is $$E[\overline V] = Var(X) (1 - \sum_j \lambda_j^2)$$ It remains to plug in the expression of $\lambda_i$ with respect to $\omega_i$ to get variant C.

-
 That's variant C above, isn't it? – Anony-Mousse Jan 10 at 7:14 Oups, yes, it is variant C. – ThePawn Jan 10 at 7:39

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer: please explain why you're recommending it as a solution. Answers that don't explain anything will be deleted. See Good Subjective, Bad Subjective for more information.