I'm looking for the correct equation to compute the weighted unbiased sample covariance. Internet sources are quite rare on this theme and they all use different equations.

The most likely equation I've found is this one:

$q_{jk}=\frac{\sum_{i=1}^{N}w_i}{\left(\sum_{i=1}^{N}w_i\right)^2-\sum_{i=1}^{N}w_i^2} \sum_{i=1}^N w_i \left( x_{ij}-\bar{x}_j \right) \left( x_{ik}-\bar{x}_k \right) .$

From: https://en.wikipedia.org/wiki/Sample_mean_and_sample_covariance#Weighted_samples

Of course, you have to compute the weighted (unbiased) sample mean beforehand.

However, I have found several other formulas like:

$q_{jk}= \frac{1}{\sum_{i=1}^N w_i)-1}\sum_{i=1}^N w_i \left( x_{ij}-\bar{x}_j \right) \left( x_{ik}-\bar{x}_k \right) .$

Or I've even seen some source codes and academic papers just using the standard covariance formula but with the weighted sample mean instead of the sample mean...

Can someone help me and shed some light?

/EDIT: my weights are simply the number of observations for a sample in the dataset, thus weights.sum() = n

  • 1
    What kinds of weights are you using? Weights can mean different things, according to the application, and the correct answer depends on their meaning. For instance, they can be shorthand for frequencies (an observation has a weight of $f$ in the dataset to reflect its occurrence $f$ times in the data) or they can be shorthand for probabilities of being in a sample (for weighted random samples, such as stratified or hierarchical samples). – whuber Jun 8 '13 at 18:57
  • @whuber: My weights are simply the number of observations for one sample in the dataset, and thus: Sum(weights) = n – gaborous Jun 8 '13 at 23:01
  • Then your question is answered in many places here (although that might not immediately be apparent), including stats.stackexchange.com/questions/58986/… (which explains the principle) as well as stats.stackexchange.com/questions/6534/… (which I still maintain is incorrect, but you can decide for yourself). – whuber Jun 9 '13 at 13:41
  • @whuber: thank you for your help, but the first link is about standard deviation (squared root of variance) and not about covariance, and the second one is plainly wrong (as you pointed). – gaborous Jun 9 '13 at 16:43
  • 2
    There is no difference between variance and covariance: all covariances can be obtained as (linear combinations) of variances and variances, of course, are just particular examples of covariances. (Mathematicians call this relationship polarization.) – whuber Jun 10 '13 at 13:01
up vote 12 down vote accepted

Found the solution in a 1972's book (George R. Price, Ann. Hum. Genet., Lond, pp485-490, Extension of covariance selection mathematics, 1972).

Biased weighted sample covariance:

$\Sigma=\frac{1}{\sum_{i=1}^{N}w_i}\sum_{i=1}^N w_i \left(x_i - \mu^*\right)^T\left(x_i - \mu^*\right)$

And the unbiased weighted sample covariance given by applying the Bessel correction:

$\Sigma=\frac{1}{\sum_{i=1}^{N}w_i - 1}\sum_{i=1}^N w_i \left(x_i - \mu^*\right)^T\left(x_i - \mu^*\right)$

Where $\mu^*$ is the (unbiased) weighted sample mean:

$\mathbf{\mu^*}=\frac{\sum_{i=1}^N w_i \mathbf{x}_i}{\sum_{i=1}^N w_i}$

Important Note: this works only if the weights are "repeat"-type weights, meaning that each weight represent the number of occurrences of one observation, and that $\sum_{i=1}^N w_i=N^*$ where $N^*$ represent the real sample size (real total number of samples, accounting for the weights).

I have updated the article on Wikipedia, where you will also find the equation for unbiased weighted sample variance:

https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_covariance

Practical note: I advise you to first multiply column-by-column $w_i$ and $\left(x_i - \mu^*\right)$ and then do a matrix multiplication with $\left(x_i - \mu^*\right)$ to wrap things up and automatically perform the summation. Eg in Python Pandas/Numpy code:

import pandas as pd
import numpy as np
# X is the dataset, as a Pandas' DataFrame
mean = mean = np.ma.average(X, axis=0, weights=weights) # Computing the weighted sample mean (fast, efficient and precise)
mean = pd.Series(mean, index=list(X.keys())) # Convert to a Pandas' Series (it's just aesthetic and more ergonomic, no differenc in computed values)
xm = X-mean # xm = X diff to mean
xm = xm.fillna(0) # fill NaN with 0 (because anyway a variance of 0 is just void, but at least it keeps the other covariance's values computed correctly))
sigma2 = 1./(w.sum()-1) * xm.mul(w, axis=0).T.dot(xm); # Compute the unbiased weighted sample covariance

Did a few sanity checks using a non-weighted dataset and an equivalent weighted dataset, and it works correctly.

  • @whuber: no it's correct, but it depends on what your "weights" are assigned to. In my case, it's the number of observations (aka "repeats"), as such this equation correctly works. In your case with normalized weights or also if the weights are the variances of each measurement of an observation, so-called "reliability", then it doesn't work and the other equation on the wiki should be used (which btw doesn't work with "repeats"-type weights!). – gaborous Jun 10 '13 at 18:15
  • @whuber: anyway if you have a more generalizable approach, I would be glad to hear about it. I have crawled the whole internet and this website and didn't find an equation that would work for repeats-type weights except this one I have posted above! – gaborous Jun 10 '13 at 18:16
  • 2
    There cannot possibly be a universal formula: if the weights are not integral frequencies (for instance, if they have been standardized to sum to unity), then you have lost all information about the total sample size $n$, whence it is impossible to estimate the correction factor $n/(n-1)$. Nevertheless, provided the weights do not sum to unity, your formula will produce an answer. That is why it's crucial to make clear that your formula is not a generally applicable one and that the $w_i$ must be actual frequencies and not anything else. – whuber Jun 10 '13 at 18:20
  • 3
    @whuber: Thank's for the explanation, that's what I feared (no unified equation and loss of correction factor). I have added a note to my answer to better describe this. I am also going to add your line about the correction factor on the Wikipedia article. – gaborous Jun 10 '13 at 18:39
  • @whuber Came across these old threads. I am wondering how your statement that "there cannot possibly be a universal formula" fits to e.g. an accepted answer in this thread: stats.stackexchange.com/questions/47325. Do you think that answer is wrong (in claiming that there is an unbiased expression for any weights)? – amoeba Feb 6 '17 at 19:50

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.