# Bias correction in weighted variance

For unweighted variance $$\text{Var}(X):=\frac{1}{n}\sum_i(x_i - \mu)^2$$ there exists the bias corrected sample variance, when the mean was estimated from the same data: $$\text{Var}(X):=\frac{1}{n-1}\sum_i(x_i - E[X])^2$$

I'm looking into weighted mean and variance, and wondering what the appropriate bias correction for the weighted variance is. Using: $$\text{mean}(X):=\frac{1}{\sum_i \omega_i}\sum_i \omega_i x_i$$

The "naive", non-corrected variance I'm using is this: $$\text{Var}(X):=\frac{1}{\sum_i \omega_i}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

So I'm wondering whether the correct way of correcting bias is

A) $$\text{Var}(X):=\frac{1}{\sum_i \omega_i - 1}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

or B) $$\text{Var}(X):=\frac{n}{n-1}\frac{1}{\sum_i \omega_i}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

or C) $$\text{Var}(X):=\frac{\sum_i \omega_i}{(\sum_i \omega_i)^2-\sum_i \omega_i^ 2}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

A) does not make sense to me when the weights are small. The normalization value could be 0 or even negative. But how about B) ($n$ is the number of observations) - is this the correct approach? Do you have some reference that shows this? I belive "Updating mean and variance estimates: an improved method", D.H.D. West, 1979 uses this. The third, C) is my interpretation of the answer to this question: https://mathoverflow.net/questions/22203/unbiased-estimate-of-the-variance-of-an-unnormalised-weighted-mean

For C) I have just realized that the denominator looks a lot like $\text{Var}(\Omega)$. Is there some general connection here? I think it does not entirely align; and obviously there is the connection that we are trying to compute the variance...

All three of them seem to "survive" the sanity check of setting all $\omega_i=1$. So which one should I used, under which premises? ''Update:'' whuber suggested to also do the sanity check with $\omega_1=\omega_2=.5$ and all remaining $\omega_i=\epsilon$ tiny. This seems to rule out A and B.

• When you consider cases where the two largest weights are equal and all the rest become vanishingly small, both (A) and (B) drop from contention (because they disagree with the known results for $n=2$). (C) appears to be an approximation; I suspect the correct factor is a much more complicated function of the weights.
– whuber
Jan 9 '13 at 15:44
• @whuber ThePawn below suggests that it is C. Do you have more detailed concerns? Jan 12 '13 at 10:35
• Solution (A) works, I have implement it in the past and can confirm from empiric tests that it gives the correct results. However, you must only use integer values for the weights and > 0. Jun 8 '13 at 23:10
• Thanks! This helped me a lot to get on the right track when the weights are for an exponential moving average! It turns out that the naive way to compute the variance actually overestimates it by a constant factor of 2, in addition to the small (1-1/n) correction that shows up analogously to the simple moving average calculation. That's a particularly crazy special case! Nov 28 '19 at 23:56
• you mean empirical mean x bar in your second equation, not population value E[X], right? Aug 26 '20 at 22:53

## 2 Answers

I went through the math and ended up with variant C:

$$Var(X) = \frac{(\sum_i \omega_i)^2}{(\sum_i \omega_i)^2 - \sum_i \omega_i^2}\overline V$$ where $\overline V$ is the non corrected variance estimation. The formula agrees with the unweighted case when all $\omega_i$ are identical. I detail the proof below:

Setting $\lambda_i = \frac{\omega_i}{\sum_i \omega_i}$, we have

$$\overline V = \sum_i \lambda_i (x_i - \sum_j \lambda_j x_j)^2$$

Expanding the inner term gives: $$(x_i - \sum_j \lambda_j x_j)^2 = x_i^2 + \sum_{j, k} \lambda_j \lambda_k x_j x_k - 2 \sum_j \lambda_j x_i x_j$$

If we take the expectation, we have that $E[x_i x_j] = Var(X)1_{i = j} + E[X]^2$, the term $E[X]$ being present in each term, it cancels out and we get:

$$E[\overline V] = Var(X) \sum_i \lambda_i (1 + \sum_j \lambda_j^2- 2 \lambda_i )$$ that is $$E[\overline V] = Var(X) (1 - \sum_j \lambda_j^2)$$ It remains to plug in the expression of $\lambda_i$ with respect to $\omega_i$ to get variant C.

• That's variant C above, isn't it? Jan 10 '13 at 7:14
• Oups, yes, it is variant C. Jan 10 '13 at 7:39
• I have checked this solution empirically and it does NOT work... The only one that does is solution (A) that I have also implemented in the past by myself, but it only works with weights being integer numbers and >= 0 Jun 8 '13 at 23:08
• This equation is wrong according to Wikipedia, Matlab, R, and others which are implementing this equation. The numerator here is squared, but it should NOT, it should be just like the (C) proposed by the OP. See en.wikipedia.org/wiki/… Jun 13 '13 at 10:38
• @rajatkhanduja I was not talking about the proof but the final derived equation (the top one in this answer). But indeed it's correct, the numerator is just squared because we multipy by V, thus the numerator ends up being unsquared. Anyway, this estimator remains biased as I explain in my answer below since it relies on "reliability"-type weights. Nov 23 '13 at 16:58

Both A and C are correct, but which one you will use depends on what kind of weights you use:

• A needs you to use "repeat"-type weights (integers counting the number of occurrences for each observation), and is unbiased.
• C needs you to use "reliability"-type weights (either normalized weights or either variances for each observation), and is biased. It can't be unbiased.

The reason why C is necessarily biased is because if you don't use "repeat"-type weights, you lose the ability to count the total number of observations (sample size), and thus you can't use a correction factor.

For more info, check the Wikipedia article that was updated recently: http://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_variance

/EDIT: the Wikipedia page may not be up-to-date anymore. There is a strong dogma around this issue it seems, so despite sources clearly stating reliability-type weights cannot be unbiased, and any practical implementation comparing side-by-side the results of calculating reliability-type weighted variance vs repeat-type showing divergent results.

If you want the original source and refs and technical details and even a practical implementation in Python, see my other posts here and here.