0
$\begingroup$

I have a $P\times K$ matrix $\mathbf X$ with $K$ random vectors as columns (with the respective means subtracted from each entry). My goal is to decorrelate the columns of $\mathbf X$ via PCA to obtain the transformed matrix $\mathbf Y$ of the same size as $\mathbf X$.

I proceed as follows: I form the $P \times P$ covariance matrix $\mathbf C=\frac{1}{K}E\big[\mathbf X \mathbf X^\top\big]$. Let the eigendecomposition of the covariance matrix be $\mathbf \Sigma=\mathbf U\mathbf \Lambda \mathbf U^T$ where $\mathbf U$ contains the eigenvectors and $\mathbf \Lambda = \text{diag}\{\lambda_i,\cdots,\lambda_{P}\}$ is the eigenvalue matrix of $\mathbf \Sigma$. The eigenvalues correspond to the variances. To decorrelate the columns of $\mathbf X$, we multiply this matrix on the left by $\mathbf \Lambda^{-1/2} \mathbf U^T$: $$\mathbf Y = \mathbf \Lambda^{-1/2} \mathbf U^T \mathbf X,$$ so that the transformed data $\mathbf Y$ will have an identity covariance matrix.

In the above process, it is not clear to me how the sizes stack up. Because $\mathbf Y$ is of size $P \times K$, I must have $\mathbf \Lambda$ of size $P \times P$. On the other hand, the eigenvalues correspond to variances of the random vectors in question. I have $K$ random vectors in the matrix $\mathbf X$. Therefore, I should get $\mathbf \Lambda$ of size $K \times K$.

What am I missing here?

Edit after @whuber's remarks: If I form a $K \times K$ covariance matrix, then the dimension of $\mathbf \Lambda^{-1/2} \mathbf U^T$ is $K \times K$. I can not, therefore, multiply $\mathbf \Lambda^{-1/2} \mathbf U^T$ to the $P \times K$ matrix $\mathbf X$.

Second edit after some more research: The eigenvalues of $\mathbf X \mathbf X^\top$ and $\mathbf X^\top \mathbf X$ are same (see this link); in the bigger matrix, rest of the eigenvalues are simply zero. Therefore, it doesn't matter which way I form the covariance matrix. I am not sure if the variances (which are eigenvalues themselves) would be same in both cases.

-ryan

$\endgroup$
  • $\begingroup$ $C$ is not the covariance matrix! The covariance matrix is found by forming the $K\times K$ matrix $X^\prime X$ ("E" is meaningless here: it applies to random variables but not to data) and dividing by $P$. $\endgroup$ – whuber Sep 8 '17 at 22:35
  • $\begingroup$ @whuber Thanks for the clarification. I meant to say I have formed a matrix of random vectors that I want to simultaneously decorrelate. I have modified my question now. If I form a $K \times K$ covariance matrix, then the dimension of $\mathbf \Lambda^{-1/2} \mathbf U^T$ is $K \times K$. I can not, therefore, multiply $\mathbf \Lambda^{-1/2} \mathbf U^T$ to the $P \times K$ matrix $\mathbf X$. $\endgroup$ – ryan80 Sep 9 '17 at 6:44
  • $\begingroup$ "On the other hand, the eigenvalues correspond to variances of the random vectors in question" -- No. I don't even understand what this could mean. What is a variance of one vector? $\endgroup$ – amoeba Sep 11 '17 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.