# how to implement law of total covariance

As I understand it, the law of total covariance indicates that the covariance of two data sets X and Y should be equal to the average of the covariances of all subsets of X and Y (eg. [X1, Y1] .....[Xn,Yn]) plus the covariance of the averages of X and Y for all subsets.

When implementing this, I can not achieve equality of the two approaches:

$cov(X,Y) \neq \left(\sum_{i=1}^n \frac{m_i}{m}cov(X_i,Y_i)\right) + cov([\overline{X_1}..\overline{X_n}], [\frac{m_1}{m}\overline{Y_1}..\frac{m_n}{m}\overline{Y_n}])$

where $m$ is total number of samples in $X$ and $Y$, $n$ is the number of subsets of $X$ and $Y$, and $m_i$ is the number of samples in the subsets $X_i$ and $Y_i$.

I assumed that the subset covariances and averages needed to be weighted by the fraction of samples in their corresponding subsets. While this approach brings the two estimates of total covariance close, the two answers are not exactly identical.

Any suggests as to what is wrong with my approach.

-------------- example of approach ----

For $X$ and $Y$ vectors, which can be separated into three subsets:

$X_1 = [x_1 .. x_{m1}] , \quad Y_1 = [y_1 .. y_{m1}] \\ X_2 = [x_1 .. x_{m2}] , \quad Y_2 = [y_1 .. y_{m2}]\\ X_3 = [x_1 .. x_{m3}] , \quad Y_3 = [y_1 .. y_{m3}]\\$

I first calculate the weighting for each subset pair:

$w1 = \frac{m1}{m1+m2+m3} \\ w2 = \frac{m2}{m1+m2+m3} \\ w3 = \frac{m3}{m1+m2+m3}$

I calculate the covariance for each pair:

$c1 = cov(X_1,Y_1) \\ c2 = cov(X_2,Y_2) \\ c3 = cov(X_3,Y_3) \\$

I then create vectors of the subset means, with one of the vectors weighted by the subset size:

$X_a = [\overline{X_1},\overline{X_2},\overline{X_3}] \\ Y_a = [w1 \:\overline{Y_1},w2 \: \overline{Y_2},w3 \: \overline{Y_3}]$

and then calculate their covariance:

$c_a = cov(X_a,Y_a)$

My understanding was that the total covariance should equal the sum of the weighted subset covariances and the covariance of the subset means:

$cov(X,Y) = (w1 \: c1 + w2 \: c2 + w3 \: c3) + ca$

• I am having difficulty understanding this version of the law you are using. Could you work out a tiny example for us? Consider, for instance, the datasets where both $X$ and $Y$ equal $0$ and $1$ (so that $n=2$). Your example would clarify how you are computing covariances, which subsets you are using, and even show us what you mean by two datasets (since covariance ordinarily applies only to a single vector-valued dataset).
– whuber
Nov 7, 2016 at 18:50
• Please register &/or merge your accounts (you can find information on how to do this in the My Account section of our help center), then you will be able to edit & comment on your own question. Nov 7, 2016 at 19:48

## 2 Answers

Recently, I had a similar problem - recover the total covariance from a list of covariances and means.

$$Cov_t(x,y) = \frac{\sum_{i=1}^{m} {(Cov_i(x,y) * (n_i-1)} + \sum_{i=1}^{m} {n_i(\mu_i^{(x)} - \mu^{(x)}) (\mu_i^{(y)} - \mu^{(y)}) }}{{N-1}},$$

where $m$ is total number of subsets, $n_i$ is the size of the subset $i$, $\mu_i^{(x)}$ and $\mu_i^{(y)}$ are the means of the variables (x) and (y) of the subset $i$, and $\mu^{(x)}$ and $\mu^{(y)}$ are the grand mean of the variables (x) and (y).

I have tested this formula in this notebook

From wikipedia, the raw form of the law of total variance is:

$$Cov(X, Y) = E[COV(X, Y|Z)] + Cov(E[X|Z], E[Y|Z])$$

In your case, the random variable $$Z$$ would represent subset assignment. I think the first term in your equation is correct, but the second term is not.

Let $$h(Z) = E[X|Z], g(Z) = E[Y|Z]$$. The key point is, both $$h(Z)$$ and $$g(Z)$$ are still random variables. To be more specific:

$$h(Z) = \sum_i^n \mathbf{1}(Z = i)\bar{X}_i = \bar{X}_i \text{with probability } \frac{m_i}{m} \quad (\text{for } i = 1, 2, \ldots)$$

Similarly,

$$g(Z) = \sum_i^n \mathbf{1}(Z = i)\bar{Y}_i = \bar{Y}_i \text{with probability } \frac{m_i}{m} \quad (\text{for } i = 1, 2, \ldots)$$

Now we can calculate the second term as:

$$Cov(h(Z), g(Z)) = E h(Z)g(Z) - E h(Z) E g(z)$$

$$= \sum_i^n \frac{m_i}{m} \bar{X}_i \bar{Y}_i - \bar{X} \bar{Y}$$

With this, you should be able to achieve equality.