3
$\begingroup$

How can I obtain the covariance matrix when all the variances of the variables are known?

This is from the paper http://www.cs.berkeley.edu/~jordan/papers/CSD-04-1330.pdf

$$V_1 \sim \mathcal{N}(0,290), V_2 \sim \mathcal{N}(0,300), V_3 = -0.3V_1 + 0.925V_2+ \epsilon, \epsilon \sim \mathcal{N}(0,300)$$ with $V_1,V_2$ and $\epsilon$ independent. Afterward, 10 observed variales are generated as follows. $$X_i = V_j + \epsilon_i^j, \epsilon_i^j\sim\mathcal{N}(0,1)$$ with $j=1$ for $i=1,\dots,4$, $j=2$ for $i=5,\dots,8$, and $j=3$ for $i=9,10$ and $\epsilon_i^j$ independent for $j=1,2,3, i=1,\dots,10$. We use the exact covariance matrix to compute principal components using the different approaches.

How can I get this exact covariance matrix?

$\endgroup$
  • 3
    $\begingroup$ Simply knowing the variances is insufficient to specify the covariance matrix. You need to stipulate the covariances for the data you want to generate. $\endgroup$ – gung - Reinstate Monica Oct 30 '15 at 0:01
  • 1
    $\begingroup$ @gung It seems to me that all the required information is already present in this question, at least if we assume the $\epsilon^j_i$ are independent of the $V_j$ and $\epsilon$. $\endgroup$ – whuber Oct 30 '15 at 2:28
  • $\begingroup$ +1. I removed your sparse and pca tags, because even though the paper is about sparse PCA, this example and your question have nothing to do with it. Covariance matrix of $X$ is $10\times 10$. What elements of this matrix do you know how to compute, and what elements not? E.g. variance of $X_1$ is $290+1$. And covariance between $X_1$ and $X_5$ is zero. Do you see this? How many elements can you fill in? $\endgroup$ – amoeba says Reinstate Monica Oct 31 '15 at 21:16
  • 2
    $\begingroup$ It appears that the original paper has a typo when defining $\epsilon$; the correct form is $\epsilon \sim N(0,1)$ *not $N(0,300)$. (See my answer below for details). $\endgroup$ – usεr11852 says Reinstate Monic Nov 1 '15 at 8:23
7
$\begingroup$

You need the following properties to calculate the exact covariance matrix.

  1. If $X_1$ and $X_2$ are independent normal random variables then their sum is also an normal random variable. $X_1 \sim N(\mu_1,\sigma_1^2), X_2 \sim N(\mu_2,\sigma_2^2)$ if $X_3 = X_1 + X_2$ then $X_3 \sim N( \mu_1+\mu_2, \sigma_1^2+ \sigma_2^2)$. This holds even more generally when $X_3 = w_1X_1 + w_2X_2$ as $X_3 \sim N( w_1\mu_1 + w_2\mu_2, w_1^2 \sigma_1^2 + w_2^2 \sigma_2^2 )$.
  2. If $X_1$ and $X_2$ are independent normal random variables, then their covariance is zero (The reverse is not true; see here for more details.).
  3. The covariance of a normal random variable $X_1$ with itself is its own variance.
  4. The covariance operator is linear, ie. the covariance $\text{Cov}(X_1, c_0 X_2) = c_0 \text{Cov}(X_1,X_2)$ and more generally $\text{Cov}(X_1, c_0 X_2 + c_1 X_3 ) = c_0 \text{Cov}(X_1,X_2) + c_1 \text{Cov}(X_1,X_3)$.

The first property allows you to calculate the diagonal elements. The diagonal elements are the variance of each respective $X_i$. So for example for $i = 9$ one would need the variance of $V_3$ plus one. The variance of $V_3$ is: \begin{align} w_1^2 \sigma_1^2 +& w_2^2 \sigma_2^2 + \sigma_\epsilon^2 = \\ (-0.3)^2 290 +& (0.925)^2 300 + 1 = 283.7875 \end{align} Based on this the variance of $X_9$ (and $X_{10}$) should be $284.7875$. [Notice that for $\sigma_\epsilon^2$ I assume a magnitude of 1 instead of 300. That is because in the paper by d’Aspremont et al. there is a typo. The original paper by Zou et al. does not have this typo. This is easy to spot. If this was not the case the original estimate of the first two components account for 99%+ of the variation could not be true.]

The second property allows you calculate the covariance between independent variables, eg. $X_1$ and $X_5$. $\text{Cov}(X_1,X_5) = 0$.

The third property allows you calculate the covariance of $X_9$ and $X_{10}$. Notice that it will not be zero; $X_9$ and $X_{10}$ are constructed using $V_3$ as a common building block and $\text{Var}(V_3) = 283.7875$. We need the covariance $\text{Cov}(V_3 + \epsilon_3^9,V_3 + \epsilon^{10}_3)$. As the covariances $\text{Cov}( \epsilon^{9}_3, \epsilon^{10}_3) $, $\text{Cov}( V_3, \epsilon^{10}_3) $ and $\text{Cov}( \epsilon^{9}_3, V_3) $ equal to zero then you are left with $\text{Cov}(V_3, V_3)$ which as we mentioned equals to the variance of $V_3$, $\text{Var}(V_3) = 283.7875$.

Finally,the fourth property (bilinearity) is what allows you to get the covariance of $X_9$ and $X_1$ (or any other $i = 1, \dots, 8$)? One needs essentially the covariance $\text{Cov}(V_1 + \epsilon_1^1,V_3 + \epsilon^{9}_3)$ which equals $\text{Cov}(V_1 + \epsilon_1^1, -0.3 V_1 + 0.925 V_2 + \epsilon + \epsilon^{9}_3)$. Expanding this and taking into account that the covariances $\text{Cov}(V_1,V_2)$ $\text{Cov}(V_1, \epsilon)$, $\text{Cov}(V_1, \epsilon^{9}_3)$, $\text{Cov}(\epsilon_1^1,V_2)$ $\text{Cov}(\epsilon_1^1, \epsilon)$ and $\text{Cov}(\epsilon_1^1, \epsilon^{9}_3)$ are all equal to zero you are left out with $\text{Cov}(V_1 + \epsilon_1^1,V_3 + \epsilon^{9}_3) = \text{Cov}(V_1, -0.3 V_1) $ which based on the linearity property is just $-0.3\text{Var}(V_1) = -87$.

That's how you get the final exact covariance matrix: \begin{bmatrix} 291 & 290 & 290 & 290 & 0 & 0 & 0 & 0 & -87 & -87 \\ 290 & 291 & 290 & 290 & 0 & 0 & 0 & 0 & -87 & -87 \\ 290 & 290 & 291 & 290 & 0 & 0 & 0 & 0 & -87 & -87 \\ 290 & 290 & 290 & 291 & 0 & 0 & 0 & 0 & -87 & -87 \\ 0 & 0 & 0 & 0 & 301 & 300 & 300 & 300 & 277.5 & 277.5\\ 0 & 0 & 0 & 0 & 300 & 301 & 300 & 300 & 277.5 & 277.5\\ 0 & 0 & 0 & 0 & 300 & 300 & 301 & 300 & 277.5 & 277.5\\ 0 & 0 & 0 & 0 & 300 & 300 & 300 & 301 & 277.5 & 277.5\\ -87 & -87 & -87 & -87 & 277.5 & 277.5& 277.5 & 277.5& 284.7875 & 283.7875 \\ -87 & -87 & -87 & -87 & 277.5 & 277.5& 277.5 & 277.5& 283.7875 & 284.7875 \\ \end{bmatrix}

which according to MATLAB has the cumulative Fraction-of-Variance-Explained (FVE) from each principal mode of variation equal to: \begin{align} \lambda = \begin{bmatrix} 0.6004\\ 0.9968\\ 0.9976\\ 0.9980\\ 0.9983\\ 0.9986\\ 0.9990\\ 0.9993\\ 0.9997\\ 1.0000\end{bmatrix} \end{align}

For the kicks I also do a simulation (in MATLAB) to make sure I get the very similar results with the paper:

rng(123)
N = 400^3;
V1 = random('normal', 0,sqrt(290), N, 1);                       
V2 = random('normal', 0,sqrt(300), N, 1);                 
V3 = -0.3*V1 + 0.925*V2 +  random('normal', 0,1, N,1);
X = [ V1+ randn(N,1), V1+ randn(N,1), V1+ randn(N,1), V1+ randn(N,1),...
      V2+ randn(N,1), V2+ randn(N,1), V2+ randn(N,1), V2+ randn(N,1),...
      V3+ randn(N,1), V3+ randn(N,1)];                             
[V,D] = eig(cov(X)); 
cumsum( flipud(diag(D))) ./ sum(diag(D))             
%ans =
%    0.6006
%    0.9968  << Yeap, it works.
%    0.9976
%    0.9980
%    0.9983
%    0.9986
%    0.9990
%    0.9993
%    0.9997
%    1.0000
$\endgroup$
  • $\begingroup$ Thank you for the great answer! Let me ask one more thing, why is "3.The covariance of a normal random variable X1 with itself is its own variance."? Is there any site I can see the proof of this? $\endgroup$ – user26767 Nov 1 '15 at 13:27
  • 1
    $\begingroup$ That assertion is trivial: the two definitions agree, regardless of the underlying distribution of $X_1$. $\endgroup$ – whuber Nov 1 '15 at 15:21
  • 1
    $\begingroup$ @user26767: As whuber mentioned this is consider an elementary result. To quote Wikipedia: "Variance is a special case of the covariance when the two variables are identical". One can easily see this by taking the standard formula for the covariance of $X_1$ and $X_2$ $E[(X_1-E[X_1])(X_2-E[X_2])]$ and substituting $X_2$ with $X_1$. The formula for the variance for a single variable comes up, $E[(X_1-E[X_1])^2]$. $\endgroup$ – usεr11852 says Reinstate Monic Nov 1 '15 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.