17
$\begingroup$

Unbiased weighted variance was already addressed here and elsewhere but there still seems to be a surprising amount of confusion. There appears to be a consensus toward the formula presented in the first link as well as in the Wikipedia article. This also looks like the formula used by R, Mathematica, and GSL (but not MATLAB). However, the Wikipedia article also contains the following line which looks like a great sanity check for a weighted variance implementation:

For example, if values {2,2,4,5,5,5} are drawn from the same distribution, then we can treat this set as an unweighted sample, or we can treat it as the weighted sample {2,4,5} with corresponding weights {2,1,3}, and we should get the same results.

My calculations give the value of 2.1667 for variance of the original values and 2.9545 for the weighted variance. Should I really expect them to be the same? Why or why not?

$\endgroup$
  • 6
    $\begingroup$ this question is not really about implementation, but the theory behind it $\endgroup$ – confusedCoder Mar 6 '13 at 16:31
15
$\begingroup$

Yes, you should expect both examples (unweighted vs weighted) to give you the same results.

I have implemented the two algorithms from the Wikipedia article.

This one works:

If all of the $x_i$ are drawn from the same distribution and the integer weights $w_i$ indicate frequency of occurrence in the sample, then the unbiased estimator of the weighted population variance is given by:

$s^2\ = \frac {1} {V_1 - 1} \sum_{i=1}^N w_i \left(x_i - \mu^*\right)^2,$

However this one (using fractional weights) does not work for me:

If each $x_i$ is drawn from a Gaussian distribution with variance $1/w_i$, the unbiased estimator of a weighted population variance is given by:

$s^2\ = \frac {V_1} {V_1^2-V_2} \sum_{i=1}^N w_i \left(x_i - \mu^*\right)^2$

I am still investigating the reasons why the second equation does not work as intended.

/EDIT: Found the reason why the second equation did not work as I thought: you can use the second equation only if you have normalized weights or variance ("reliability") weights, and it is NOT unbiased, because if you don't use "repeat" weights (counting the number of times an observation was observed and thus should be repeated in your math operations), you lose the ability to count the total number of observations, and thus you can't use a correction factor.

So this explains the difference in your results using weighted and non-weighted variance: your computation is biased.

Thus, if you want to have an unbiased weighted variance, use only "repeat" weights and use the first equation I have posted above. If that's not possible, well, you can't help it.

I have also updated the Wikipedia's article if you want more info: http://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_variance

And a linked article about unbiased weighted covariance (which in fact is the same variance due to Polarization Identity): Correct equation for weighted unbiased sample covariance

$\endgroup$
  • $\begingroup$ After reading and thinking a lot through this I still don't get an intuitive meaning or example of the term "reliability weights". Can you please elaborate a bit on that? $\endgroup$ – Peter Aug 31 '17 at 13:58
  • $\begingroup$ @Peter reliability weights are normalized weights, eg, bounded between 0 and 1 or -1 and 1. They represent a frequency (eg, 0.1 means that this sample was seen 10% of the time compared to all other samples). I did not invent the term, it can be found in publications. For repeat weights it is the opposite, each weight represent the number of occurences, the cardinality (eg, 10 if the sample was observed 10 times). $\endgroup$ – gaborous Sep 1 '17 at 14:06
  • $\begingroup$ This is confusing because what you call repeat weights is often also called frequency weights, but I think I got the difference. It depends on normalization, right? $\endgroup$ – Peter Sep 3 '17 at 9:33
  • $\begingroup$ No, frequency weights is an alternative name for reliability weights. For repeat weights, it's the number of occurences, not the frequency. With repeat weights, there is no normalization at all, that's the point: as long as you normalize your weights, you lose the base frequency, so you cannot totally unbias your calculations. The only way is to keep the total number of occurences. If you really want to use frequency weights, I think if you store beforehand the total N number of occurences you can convert back and forth to repeat weights by multiplying frequency weights by N, then that's OK. $\endgroup$ – gaborous Sep 6 '17 at 18:45
  • $\begingroup$ And if your weights are 1/variance weights, how would you call those? Would that be "reliability weights" then? $\endgroup$ – Tom Wenseleers Jun 18 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.