Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

For unweighted variance $$\text{Var}(X):=\frac{1}{n}\sum_i(x_i - \mu)^2$$ there exists the bias corrected sample variance, when the mean was estimated from the same data: $$\text{Var}(X):=\frac{1}{n-1}\sum_i(x_i - E[X])^2$$

I'm looking into weighted mean and variance, and wondering what the appropriate bias correction for the weighted variance is. Using: $$\text{mean}(X):=\frac{1}{\sum_i \omega_i}\sum_i \omega_i x_i$$

The "naive", non-corrected variance I'm using is this: $$\text{Var}(X):=\frac{1}{\sum_i \omega_i}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

So I'm wondering whether the correct way of correcting bias is

A) $$\text{Var}(X):=\frac{1}{\sum_i \omega_i - 1}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

or B) $$\text{Var}(X):=\frac{n}{n-1}\frac{1}{\sum_i \omega_i}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

or C) $$\text{Var}(X):=\frac{\sum_i \omega_i}{(\sum_i \omega_i)^2-\sum_i \omega_i^ 2}\sum_i\omega_i(x_i - \text{mean}(X))^2$$

A) does not make sense to me when the weights are small. The normalization value could be 0 or even negative. But how about B) ($n$ is the number of observations) - is this the correct approach? Do you have some reference that shows this? I belive "Updating mean and variance estimates: an improved method", D.H.D. West, 1979 uses this. The third, C) is my interpretation of the answer to this question: http://mathoverflow.net/questions/22203/unbiased-estimate-of-the-variance-of-an-unnormalised-weighted-mean

For C) I have just realized that the denominator looks a lot like $\text{Var}(\Omega)$. Is there some general connection here? I think it does not entirely align; and obviously there is the connection that we are trying to compute the variance...

All three of them seem to "survive" the sanity check of setting all $\omega_i=1$. So which one should I used, under which premises? ''Update:'' whuber suggested to also do the sanity check with $\omega_1=\omega_2=.5$ and all remaining $\omega_i=\epsilon$ tiny. This seems to rule out A and B.

share|improve this question
    
When you consider cases where the two largest weights are equal and all the rest become vanishingly small, both (A) and (B) drop from contention (because they disagree with the known results for $n=2$). (C) appears to be an approximation; I suspect the correct factor is a much more complicated function of the weights. –  whuber Jan 9 '13 at 15:44
    
@whuber ThePawn below suggests that it is C. Do you have more detailed concerns? –  Anony-Mousse Jan 12 '13 at 10:35
    
Solution (A) works, I have implement it in the past and can confirm from empiric tests that it gives the correct results. However, you must only use integer values for the weights and > 0. –  user1121352 Jun 8 '13 at 23:10

2 Answers 2

up vote 6 down vote accepted

I went through the math and ended up with variant C:

$$Var(X) = \frac{(\sum_i \omega_i)^2}{(\sum_i \omega_i)^2 - \sum_i \omega_i^2}\overline V$$ where $\overline V$ is the non corrected variance estimation. The formula agrees with the unweighted case when all $\omega_i$ are identical. I detail the proof below:

Setting $\lambda_i = \frac{\omega_i}{\sum_i \omega_i}$, we have

$$\overline V = \sum_i \lambda_i (x_i - \sum_j \lambda_j x_j)^2$$

Expanding the inner term gives: $$(x_i - \sum_j \lambda_j x_j)^2 = x_i^2 + \sum_{j, k} \lambda_j \lambda_k x_j x_k - 2 \sum_j \lambda_j x_i x_j $$

If we take the expectation, we have that $E[x_i x_j] = Var(X)1_{i = j} + E[X]^2$, the term $E[X]$ being present in each term, it cancels out and we get:

$$E[\overline V] = Var(X) \sum_i \lambda_i (1 + \sum_j \lambda_j^2- 2 \lambda_i )$$ that is $$E[\overline V] = Var(X) (1 - \sum_j \lambda_j^2)$$ It remains to plug in the expression of $\lambda_i$ with respect to $\omega_i$ to get variant C.

share|improve this answer
    
That's variant C above, isn't it? –  Anony-Mousse Jan 10 '13 at 7:14
    
Oups, yes, it is variant C. –  ThePawn Jan 10 '13 at 7:39
    
I have checked this solution empirically and it does NOT work... The only one that does is solution (A) that I have also implemented in the past by myself, but it only works with weights being integer numbers and >= 0 –  user1121352 Jun 8 '13 at 23:08
1  
This equation is wrong according to Wikipedia, Matlab, R, and others which are implementing this equation. The numerator here is squared, but it should NOT, it should be just like the (C) proposed by the OP. See en.wikipedia.org/wiki/… –  user1121352 Jun 13 '13 at 10:38
    
@user1121352, the variant (C) and this equation are the same. Observe the meaning of [\lambda] in this equation, it is not just the weight, but normalized weight. Plugging in the value of [\lambda], we get the same result as variant (C). –  rajatkhanduja Nov 22 '13 at 6:44

Both A and C are correct, but which one you will use depends on what kind of weights you use:

  • A needs you to use "repeat"-type weights (integers counting the number of occurrences for each observation), and is unbiased.
  • C needs you to use "reliability"-type weights (either normalized weights or either variances for each observation), and is biased. It can't be unbiased.

The reason why C is necessarily biased is because if you don't use "repeat"-type weights, you lose the ability to count the total number of observations (sample size), and thus you can't use a correction factor.

For more info, check the Wikipedia article that was updated recently: http://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_variance

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.