Computing covariance matrix from the given variances?

Question

How can I obtain the covariance matrix when all the variances of the variables are known?

This is from the paper http://www.cs.berkeley.edu/~jordan/papers/CSD-04-1330.pdf

$$V_1 \sim \mathcal{N}(0,290), V_2 \sim \mathcal{N}(0,300), V_3 = -0.3V_1 + 0.925V_2+ \epsilon, \epsilon \sim \mathcal{N}(0,300)$$ with $V_1,V_2$ and $\epsilon$ independent. Afterward, 10 observed variales are generated as follows. $$X_i = V_j + \epsilon_i^j, \epsilon_i^j\sim\mathcal{N}(0,1)$$ with $j=1$ for $i=1,\dots,4$, $j=2$ for $i=5,\dots,8$, and $j=3$ for $i=9,10$ and $\epsilon_i^j$ independent for $j=1,2,3, i=1,\dots,10$. We use the exact covariance matrix to compute principal components using the different approaches.

How can I get this exact covariance matrix?

Answer 1

You need the following properties to calculate the exact covariance matrix.

1. If $X_1$ and $X_2$ are independent normal random variables then their sum is also an normal random variable. $X_1 \sim N(\mu_1,\sigma_1^2), X_2 \sim N(\mu_2,\sigma_2^2)$ if $X_3 = X_1 + X_2$ then $X_3 \sim N( \mu_1+\mu_2, \sigma_1^2+ \sigma_2^2)$. This holds even more generally when $X_3 = w_1X_1 + w_2X_2$ as $X_3 \sim N( w_1\mu_1 + w_2\mu_2, w_1^2 \sigma_1^2 + w_2^2 \sigma_2^2 )$.
2. If $X_1$ and $X_2$ are independent normal random variables, then their covariance is zero (The reverse is not true; see here for more details.).
3. The covariance of a normal random variable $X_1$ with itself is its own variance.
4. The covariance operator is linear, ie. the covariance $\text{Cov}(X_1, c_0 X_2) = c_0 \text{Cov}(X_1,X_2)$ and more generally $\text{Cov}(X_1, c_0 X_2 + c_1 X_3 ) = c_0 \text{Cov}(X_1,X_2) + c_1 \text{Cov}(X_1,X_3)$.

The first property allows you to calculate the diagonal elements. The diagonal elements are the variance of each respective $X_i$. So for example for $i = 9$ one would need the variance of $V_3$ plus one. The variance of $V_3$ is: \begin{align} w_1^2 \sigma_1^2 +& w_2^2 \sigma_2^2 + \sigma_\epsilon^2 = \\ (-0.3)^2 290 +& (0.925)^2 300 + 1 = 283.7875 \end{align} Based on this the variance of $X_9$ (and $X_{10}$) should be $284.7875$. [Notice that for $\sigma_\epsilon^2$ I assume a magnitude of 1 instead of 300. That is because in the paper by d’Aspremont et al. there is a typo. The original paper by Zou et al. does not have this typo. This is easy to spot. If this was not the case the original estimate of the first two components account for 99%+ of the variation could not be true.]

The second property allows you calculate the covariance between independent variables, eg. $X_1$ and $X_5$. $\text{Cov}(X_1,X_5) = 0$.

The third property allows you calculate the covariance of $X_9$ and $X_{10}$. Notice that it will not be zero; $X_9$ and $X_{10}$ are constructed using $V_3$ as a common building block and $\text{Var}(V_3) = 283.7875$. We need the covariance $\text{Cov}(V_3 + \epsilon_3^9,V_3 + \epsilon^{10}_3)$. As the covariances $\text{Cov}( \epsilon^{9}_3, \epsilon^{10}_3) $, $\text{Cov}( V_3, \epsilon^{10}_3) $ and $\text{Cov}( \epsilon^{9}_3, V_3) $ equal to zero then you are left with $\text{Cov}(V_3, V_3)$ which as we mentioned equals to the variance of $V_3$, $\text{Var}(V_3) = 283.7875$.

Finally,the fourth property (bilinearity) is what allows you to get the covariance of $X_9$ and $X_1$ (or any other $i = 1, \dots, 8$)? One needs essentially the covariance $\text{Cov}(V_1 + \epsilon_1^1,V_3 + \epsilon^{9}_3)$ which equals $\text{Cov}(V_1 + \epsilon_1^1, -0.3 V_1 + 0.925 V_2 + \epsilon + \epsilon^{9}_3)$. Expanding this and taking into account that the covariances $\text{Cov}(V_1,V_2)$ $\text{Cov}(V_1, \epsilon)$, $\text{Cov}(V_1, \epsilon^{9}_3)$, $\text{Cov}(\epsilon_1^1,V_2)$ $\text{Cov}(\epsilon_1^1, \epsilon)$ and $\text{Cov}(\epsilon_1^1, \epsilon^{9}_3)$ are all equal to zero you are left out with $\text{Cov}(V_1 + \epsilon_1^1,V_3 + \epsilon^{9}_3) = \text{Cov}(V_1, -0.3 V_1) $ which based on the linearity property is just $-0.3\text{Var}(V_1) = -87$.

That's how you get the final exact covariance matrix: \begin{bmatrix} 291 & 290 & 290 & 290 & 0 & 0 & 0 & 0 & -87 & -87 \\ 290 & 291 & 290 & 290 & 0 & 0 & 0 & 0 & -87 & -87 \\ 290 & 290 & 291 & 290 & 0 & 0 & 0 & 0 & -87 & -87 \\ 290 & 290 & 290 & 291 & 0 & 0 & 0 & 0 & -87 & -87 \\ 0 & 0 & 0 & 0 & 301 & 300 & 300 & 300 & 277.5 & 277.5\\ 0 & 0 & 0 & 0 & 300 & 301 & 300 & 300 & 277.5 & 277.5\\ 0 & 0 & 0 & 0 & 300 & 300 & 301 & 300 & 277.5 & 277.5\\ 0 & 0 & 0 & 0 & 300 & 300 & 300 & 301 & 277.5 & 277.5\\ -87 & -87 & -87 & -87 & 277.5 & 277.5& 277.5 & 277.5& 284.7875 & 283.7875 \\ -87 & -87 & -87 & -87 & 277.5 & 277.5& 277.5 & 277.5& 283.7875 & 284.7875 \\ \end{bmatrix}

which according to MATLAB has the cumulative Fraction-of-Variance-Explained (FVE) from each principal mode of variation equal to: \begin{align} \lambda = \begin{bmatrix} 0.6004\\ 0.9968\\ 0.9976\\ 0.9980\\ 0.9983\\ 0.9986\\ 0.9990\\ 0.9993\\ 0.9997\\ 1.0000\end{bmatrix} \end{align}

For the kicks I also do a simulation (in MATLAB) to make sure I get the very similar results with the paper:

rng(123)
N = 400^3;
V1 = random('normal', 0,sqrt(290), N, 1);                       
V2 = random('normal', 0,sqrt(300), N, 1);                 
V3 = -0.3*V1 + 0.925*V2 +  random('normal', 0,1, N,1);
X = [ V1+ randn(N,1), V1+ randn(N,1), V1+ randn(N,1), V1+ randn(N,1),...
      V2+ randn(N,1), V2+ randn(N,1), V2+ randn(N,1), V2+ randn(N,1),...
      V3+ randn(N,1), V3+ randn(N,1)];                             
[V,D] = eig(cov(X)); 
cumsum( flipud(diag(D))) ./ sum(diag(D))             
%ans =
%    0.6006
%    0.9968  << Yeap, it works.
%    0.9976
%    0.9980
%    0.9983
%    0.9986
%    0.9990
%    0.9993
%    0.9997
%    1.0000