how to implement law of total covariance

Question

As I understand it, the law of total covariance indicates that the covariance of two data sets X and Y should be equal to the average of the covariances of all subsets of X and Y (eg. [X1, Y1] .....[Xn,Yn]) plus the covariance of the averages of X and Y for all subsets.

When implementing this, I can not achieve equality of the two approaches:

$cov(X,Y) \neq \left(\sum_{i=1}^n \frac{m_i}{m}cov(X_i,Y_i)\right) + cov([\overline{X_1}..\overline{X_n}], [\frac{m_1}{m}\overline{Y_1}..\frac{m_n}{m}\overline{Y_n}]) $

where $m$ is total number of samples in $X$ and $Y$, $n$ is the number of subsets of $X$ and $Y$, and $m_i$ is the number of samples in the subsets $X_i$ and $Y_i$.

I assumed that the subset covariances and averages needed to be weighted by the fraction of samples in their corresponding subsets. While this approach brings the two estimates of total covariance close, the two answers are not exactly identical.

Any suggests as to what is wrong with my approach.

-------------- example of approach ----

For $X$ and $Y$ vectors, which can be separated into three subsets:

$X_1 = [x_1 .. x_{m1}] , \quad Y_1 = [y_1 .. y_{m1}] \\ X_2 = [x_1 .. x_{m2}] , \quad Y_2 = [y_1 .. y_{m2}]\\ X_3 = [x_1 .. x_{m3}] , \quad Y_3 = [y_1 .. y_{m3}]\\$

I first calculate the weighting for each subset pair:

$ w1 = \frac{m1}{m1+m2+m3} \\ w2 = \frac{m2}{m1+m2+m3} \\ w3 = \frac{m3}{m1+m2+m3} $

I calculate the covariance for each pair:

$ c1 = cov(X_1,Y_1) \\ c2 = cov(X_2,Y_2) \\ c3 = cov(X_3,Y_3) \\ $

I then create vectors of the subset means, with one of the vectors weighted by the subset size:

$ X_a = [\overline{X_1},\overline{X_2},\overline{X_3}] \\ Y_a = [w1 \:\overline{Y_1},w2 \: \overline{Y_2},w3 \: \overline{Y_3}] $

and then calculate their covariance:

$ c_a = cov(X_a,Y_a) $

My understanding was that the total covariance should equal the sum of the weighted subset covariances and the covariance of the subset means:

$ cov(X,Y) = (w1 \: c1 + w2 \: c2 + w3 \: c3) + ca $

Answer 1

Recently, I had a similar problem - recover the total covariance from a list of covariances and means.

$$Cov_t(x,y) = \frac{\sum_{i=1}^{m} {(Cov_i(x,y) * (n_i-1)} + \sum_{i=1}^{m} {n_i(\mu_i^{(x)} - \mu^{(x)}) (\mu_i^{(y)} - \mu^{(y)}) }}{{N-1}}, $$

where $m$ is total number of subsets, $n_i$ is the size of the subset $i$, $\mu_i^{(x)}$ and $\mu_i^{(y)}$ are the means of the variables (x) and (y) of the subset $i$, and $\mu^{(x)}$ and $\mu^{(y)}$ are the grand mean of the variables (x) and (y).

I have tested this formula in this notebook

Answer 2

From wikipedia, the raw form of the law of total variance is:

$Cov(X, Y) = E[COV(X, Y|Z)] + Cov(E[X|Z], E[Y|Z])$

In your case, the random variable $Z$ would represent subset assignment. I think the first term in your equation is correct, but the second term is not.

Let $h(Z) = E[X|Z], g(Z) = E[Y|Z]$. The key point is, both $h(Z)$ and $g(Z)$ are still random variables. To be more specific:

$h(Z) = \sum_i^n \mathbf{1}(Z = i)\bar{X}_i = \bar{X}_i \text{with probability } \frac{m_i}{m} \quad (\text{for } i = 1, 2, \ldots)$

Similarly,

$g(Z) = \sum_i^n \mathbf{1}(Z = i)\bar{Y}_i = \bar{Y}_i \text{with probability } \frac{m_i}{m} \quad (\text{for } i = 1, 2, \ldots)$

Now we can calculate the second term as:

$Cov(h(Z), g(Z)) = E h(Z)g(Z) - E h(Z) E g(z)$

$= \sum_i^n \frac{m_i}{m} \bar{X}_i \bar{Y}_i - \bar{X} \bar{Y}$

With this, you should be able to achieve equality.