11
$\begingroup$

I'm looking for the correct equation to compute the weighted unbiased sample covariance. Internet sources are quite rare on this theme and they all use different equations.

The most likely equation I've found is this one:

$q_{jk}=\frac{\sum_{i=1}^{N}w_i}{\left(\sum_{i=1}^{N}w_i\right)^2-\sum_{i=1}^{N}w_i^2} \sum_{i=1}^N w_i \left( x_{ij}-\bar{x}_j \right) \left( x_{ik}-\bar{x}_k \right) .$

From: https://en.wikipedia.org/wiki/Sample_mean_and_sample_covariance#Weighted_samples

Of course, you have to compute the weighted (unbiased) sample mean beforehand.

However, I have found several other formulas like:

$q_{jk}= \frac{1}{\sum_{i=1}^N w_i)-1}\sum_{i=1}^N w_i \left( x_{ij}-\bar{x}_j \right) \left( x_{ik}-\bar{x}_k \right) .$

Or I've even seen some source codes and academic papers just using the standard covariance formula but with the weighted sample mean instead of the sample mean...

Can someone help me and shed some light?

/EDIT: my weights are simply the number of observations for a sample in the dataset, thus weights.sum() = n

$\endgroup$
9
  • 1
    $\begingroup$ What kinds of weights are you using? Weights can mean different things, according to the application, and the correct answer depends on their meaning. For instance, they can be shorthand for frequencies (an observation has a weight of $f$ in the dataset to reflect its occurrence $f$ times in the data) or they can be shorthand for probabilities of being in a sample (for weighted random samples, such as stratified or hierarchical samples). $\endgroup$
    – whuber
    Jun 8 '13 at 18:57
  • $\begingroup$ @whuber: My weights are simply the number of observations for one sample in the dataset, and thus: Sum(weights) = n $\endgroup$
    – gaborous
    Jun 8 '13 at 23:01
  • 1
    $\begingroup$ Then your question is answered in many places here (although that might not immediately be apparent), including stats.stackexchange.com/questions/58986/… (which explains the principle) as well as stats.stackexchange.com/questions/6534/… (which I still maintain is incorrect, but you can decide for yourself). $\endgroup$
    – whuber
    Jun 9 '13 at 13:41
  • $\begingroup$ @whuber: thank you for your help, but the first link is about standard deviation (squared root of variance) and not about covariance, and the second one is plainly wrong (as you pointed). $\endgroup$
    – gaborous
    Jun 9 '13 at 16:43
  • 3
    $\begingroup$ There is no difference between variance and covariance: all covariances can be obtained as (linear combinations) of variances and variances, of course, are just particular examples of covariances. (Mathematicians call this relationship polarization.) $\endgroup$
    – whuber
    Jun 10 '13 at 13:01
22
$\begingroup$

Found the solution in a 1972's book (George R. Price, Ann. Hum. Genet., Lond, pp485-490, Extension of covariance selection mathematics, 1972).

Biased weighted sample covariance:

$\Sigma=\frac{1}{\sum_{i=1}^{N}w_i}\sum_{i=1}^N w_i \left(x_i - \mu^*\right)^T\left(x_i - \mu^*\right)$

And the unbiased weighted sample covariance given by applying the Bessel correction:

$\Sigma=\frac{1}{\sum_{i=1}^{N}w_i - 1}\sum_{i=1}^N w_i \left(x_i - \mu^*\right)^T\left(x_i - \mu^*\right)$

Where $\mu^*$ is the (unbiased) weighted sample mean:

$\mathbf{\mu^*}=\frac{\sum_{i=1}^N w_i \mathbf{x}_i}{\sum_{i=1}^N w_i}$

Important Note: this works only if the weights are "repeat"-type weights, meaning that each weight represent the number of occurrences of one observation, and that $\sum_{i=1}^N w_i=N^*$ where $N^*$ represent the real sample size (real total number of samples, accounting for the weights).

I have updated the article on Wikipedia, where you will also find the equation for unbiased weighted sample variance:

https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_covariance

Practical note: I advise you to first multiply column-by-column $w_i$ and $\left(x_i - \mu^*\right)$ and then do a matrix multiplication with $\left(x_i - \mu^*\right)$ to wrap things up and automatically perform the summation. Eg in Python Pandas/Numpy code:

import pandas as pd
import numpy as np
# X is the dataset, as a Pandas' DataFrame
mean = mean = np.ma.average(X, axis=0, weights=weights) # Computing the weighted sample mean (fast, efficient and precise)
mean = pd.Series(mean, index=list(X.keys())) # Convert to a Pandas' Series (it's just aesthetic and more ergonomic, no differenc in computed values)
xm = X-mean # xm = X diff to mean
xm = xm.fillna(0) # fill NaN with 0 (because anyway a variance of 0 is just void, but at least it keeps the other covariance's values computed correctly))
sigma2 = 1./(w.sum()-1) * xm.mul(w, axis=0).T.dot(xm); # Compute the unbiased weighted sample covariance

Did a few sanity checks using a non-weighted dataset and an equivalent weighted dataset, and it works correctly.

For more details about the theory of unbiased variance/covariance, see this post.

$\endgroup$
8
  • 3
    $\begingroup$ There cannot possibly be a universal formula: if the weights are not integral frequencies (for instance, if they have been standardized to sum to unity), then you have lost all information about the total sample size $n$, whence it is impossible to estimate the correction factor $n/(n-1)$. Nevertheless, provided the weights do not sum to unity, your formula will produce an answer. That is why it's crucial to make clear that your formula is not a generally applicable one and that the $w_i$ must be actual frequencies and not anything else. $\endgroup$
    – whuber
    Jun 10 '13 at 18:20
  • 3
    $\begingroup$ @whuber: Thank's for the explanation, that's what I feared (no unified equation and loss of correction factor). I have added a note to my answer to better describe this. I am also going to add your line about the correction factor on the Wikipedia article. $\endgroup$
    – gaborous
    Jun 10 '13 at 18:39
  • 1
    $\begingroup$ @Amoeba I believe the controversy played out in the comments makes my point well: there are different kinds of weights that call for different formulas. $\endgroup$
    – whuber
    Feb 6 '17 at 20:00
  • 1
    $\begingroup$ @whuber: What is your take on the formula that is used by numpy.org/doc/stable/reference/generated/numpy.cov.html? Isn't that general? $\endgroup$
    – Make42
    Apr 30 at 15:06
  • 2
    $\begingroup$ @Make42 It is not general. There are different kinds of weights that can enter into formulas differently. Some weights adjust for different levels of imprecision (variance weights); others for different inclusion probabilities in a sample (probability weights); as a shorthand for specifying multiple observations (frequency weights); and yet others for inducing arbitrary weighting to give greater importance to specified observations in a dataset. $\endgroup$
    – whuber
    Apr 30 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.