Bias correction in weighted variance

Question

For unweighted variance $$\text{Var}(X):=\frac{1}{n}\sum_i(x_i - \mu)^2$$ there exists the bias corrected sample variance, when the mean was estimated from the same data: $$\text{Var}(X):=\frac{1}{n-1}\sum_i(x_i - E[X])^2$$

I'm looking into weighted mean and variance, and wondering what the appropriate bias correction for the weighted variance is. Using: $$\text{mean}(X):=\frac{1}{\sum_i \omega_i}\sum_i \omega_i x_i$$

The "naive", non-corrected variance I'm using is this: $$\text{Var}(X):=\frac{1}{\sum_i \omega_i}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

So I'm wondering whether the correct way of correcting bias is

A) $$\text{Var}(X):=\frac{1}{\sum_i \omega_i - 1}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

or B) $$\text{Var}(X):=\frac{n}{n-1}\frac{1}{\sum_i \omega_i}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

or C) $$\text{Var}(X):=\frac{\sum_i \omega_i}{(\sum_i \omega_i)^2-\sum_i \omega_i^ 2}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

A) does not make sense to me when the weights are small. The normalization value could be 0 or even negative. But how about B) ($n$ is the number of observations) - is this the correct approach? Do you have some reference that shows this? I belive "Updating mean and variance estimates: an improved method", D.H.D. West, 1979 uses this. The third, C) is my interpretation of the answer to this question: https://mathoverflow.net/questions/22203/unbiased-estimate-of-the-variance-of-an-unnormalised-weighted-mean

For C) I have just realized that the denominator looks a lot like $\text{Var}(\Omega)$. Is there some general connection here? I think it does not entirely align; and obviously there is the connection that we are trying to compute the variance...

All three of them seem to "survive" the sanity check of setting all $\omega_i=1$. So which one should I used, under which premises? ''Update:'' whuber suggested to also do the sanity check with $\omega_1=\omega_2=.5$ and all remaining $\omega_i=\epsilon$ tiny. This seems to rule out A and B.

Answer 1

I went through the math and ended up with variant C:

$$Var(X) = \frac{(\sum_i \omega_i)^2}{(\sum_i \omega_i)^2 - \sum_i \omega_i^2}\overline V$$ where $\overline V$ is the non corrected variance estimation. The formula agrees with the unweighted case when all $\omega_i$ are identical. I detail the proof below:

Setting $\lambda_i = \frac{\omega_i}{\sum_i \omega_i}$, we have

$$\overline V = \sum_i \lambda_i (x_i - \sum_j \lambda_j x_j)^2$$

Expanding the inner term gives: $$(x_i - \sum_j \lambda_j x_j)^2 = x_i^2 + \sum_{j, k} \lambda_j \lambda_k x_j x_k - 2 \sum_j \lambda_j x_i x_j $$

If we take the expectation, we have that $E[x_i x_j] = Var(X)1_{i = j} + E[X]^2$, the term $E[X]$ being present in each term, it cancels out and we get:

$$E[\overline V] = Var(X) \sum_i \lambda_i (1 + \sum_j \lambda_j^2- 2 \lambda_i )$$ that is $$E[\overline V] = Var(X) (1 - \sum_j \lambda_j^2)$$ It remains to plug in the expression of $\lambda_i$ with respect to $\omega_i$ to get variant C.

Answer 2

Both A and C are correct, but which one you will use depends on what kind of weights you use:

• A needs you to use "repeat"-type weights (integers counting the number of occurrences for each observation), and is unbiased.
• C needs you to use "reliability"-type weights (either normalized weights or either variances for each observation), and is biased. It can't be unbiased.

The reason why C is necessarily biased is because if you don't use "repeat"-type weights, you lose the ability to count the total number of observations (sample size), and thus you can't use a correction factor.

For more info, check the Wikipedia article that was updated recently: http://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_variance

/EDIT: the Wikipedia page may not be up-to-date anymore. There is a strong dogma around this issue it seems, so despite sources clearly stating reliability-type weights cannot be unbiased, and any practical implementation comparing side-by-side the results of calculating reliability-type weighted variance vs repeat-type showing divergent results.

If you want the original source and refs and technical details and even a practical implementation in Python, see my other posts here and here.