27
$\begingroup$

Unbiased weighted variance was already addressed here and elsewhere but there still seems to be a surprising amount of confusion. There appears to be a consensus toward the formula presented in the first link as well as in the Wikipedia article. This also looks like the formula used by R, Mathematica, and GSL (but not MATLAB). However, the Wikipedia article also contains the following line which looks like a great sanity check for a weighted variance implementation:

For example, if values {2,2,4,5,5,5} are drawn from the same distribution, then we can treat this set as an unweighted sample, or we can treat it as the weighted sample {2,4,5} with corresponding weights {2,1,3}, and we should get the same results.

My calculations give the value of 2.1667 for variance of the original values and 2.9545 for the weighted variance. Should I really expect them to be the same? Why or why not?

$\endgroup$
1
  • 7
    $\begingroup$ this question is not really about implementation, but the theory behind it $\endgroup$ Commented Mar 6, 2013 at 16:31

1 Answer 1

23
$\begingroup$

Yes, you should expect both examples (unweighted vs weighted) to give you the same results.

I have implemented the two algorithms from the Wikipedia article.

This one works:

If all of the $x_i$ are drawn from the same distribution and the integer weights $w_i$ indicate frequency of occurrence in the sample, then the unbiased estimator of the weighted population variance is given by:

$s^2\ = \frac {1} {V_1 - 1} \sum_{i=1}^N w_i \left(x_i - \mu^*\right)^2,$

However this one (using fractional weights) does not work for me:

If each $x_i$ is drawn from a Gaussian distribution with variance $1/w_i$, the unbiased estimator of a weighted population variance is given by:

$s^2\ = \frac {V_1} {V_1^2-V_2} \sum_{i=1}^N w_i \left(x_i - \mu^*\right)^2$

Where $V_{1}=\sum _{i=1}^{N}w_{i}$ and $V_{2}=\sum _{i=1}^{N}w_{i}^{2}$

I am still investigating the reasons why the second equation does not work as intended.

/EDIT: Found the reason why the second equation did not work as I thought: you can use the second equation only if you have normalized weights or variance ("probability/reliability") weights, and it is NOT unbiased, because if you don't use "occurrences/repeat" weights (counting the number of times an observation was observed and thus should be repeated in your math operations), you lose the ability to count the total number of observations, and thus you can't use a correction factor.

So this explains the difference in your results using weighted and non-weighted variance: your computation is biased.

Thus, if you want to have an unbiased weighted variance, use only "occurrences/repeat" weights and use the first equation I have posted above. If that's not possible, well, you can't help it.

For more theoretical details, here is another post about unbiased weighted covariance with a reference about why we cannot unbias with probability/reliability type weights and a python implementation.

/EDIT a few years later: there is still some confusion as to why we cannot unbias probability/reliability weights.

First, to clarify, the difference between probability/reliability weights and repeat/occurrences weights is that probability/reliability weights are normalized, whereas repeat/occurrences weights are not, so you can get the total number of occurrences by just summing the latter but not the former. This is necessary to unbias because otherwise you lose the ability to know what I would call the statistical magnitude, what other calls polarization.

Indeed, it's like anything else in statistics: if I say that 10% of my subpopulation have X disease, what does it mean for the broader population? Well it depends on what is my subpopulation: if it's only 100 people, then my 10% figure doesn't mean much. But if it's 1 million people, then it may faithfully represent the whole population. Here it's the same, if we don't know the total N, we can't know how representative of the whole population our metric is, and hence we cannot unbias. Unbiasing is exactly the process of generalizing to the broader population.

$\endgroup$
13
  • $\begingroup$ After reading and thinking a lot through this I still don't get an intuitive meaning or example of the term "reliability weights". Can you please elaborate a bit on that? $\endgroup$
    – Peter
    Commented Aug 31, 2017 at 13:58
  • 1
    $\begingroup$ No, frequency weights is an alternative name for reliability weights. For repeat weights, it's the number of occurences, not the frequency. With repeat weights, there is no normalization at all, that's the point: as long as you normalize your weights, you lose the base frequency, so you cannot totally unbias your calculations. The only way is to keep the total number of occurences. If you really want to use frequency weights, I think if you store beforehand the total N number of occurences you can convert back and forth to repeat weights by multiplying frequency weights by N, then that's OK. $\endgroup$
    – gaborous
    Commented Sep 6, 2017 at 18:45
  • 2
    $\begingroup$ Any references for these terms? Because it sounds like it should be reliability weights vs frequency/repeat/occurrences weights. $\endgroup$
    – Tal Galili
    Commented Jun 6, 2021 at 10:12
  • 3
    $\begingroup$ I agree with @TalGalili that frequency weights and repeat/case weights are the same thing in the most common terminology. See for example stats.oarc.ucla.edu/other/mult-pkg/faq/…, stata.com/help.cgi?weight, and en.wikipedia.org/wiki/…. $\endgroup$ Commented Jan 20, 2022 at 21:44
  • 1
    $\begingroup$ This is a dumb question, but what are V1 and V2 in your equations? $\endgroup$
    – Zach
    Commented Jul 21 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.