0
$\begingroup$

This question already has an answer here:

I understood the "technique" for doing SVD and PCA.

However, I couldn't understand these two claims:

  1. PCA/SVD decorrelate the variables
  2. They do so using orthogonal transformations

For 1, PCA does the eigen decomposition of covariance matrix and SVD operates on the data matrix. How to explain that they decorrelate the variables?

In 2, I think I am missing some matrix transformation properties. How to see the transformations involved in the techniques of PCA/SVD and why are they only orthogonal.

Many thanks.

$\endgroup$

marked as duplicate by amoeba, SmallChess, Firebug, Michael Chernick, Sean Easter Apr 25 '17 at 13:43

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Your questions are somewhat vague. There is a lot of information on SVD/PCA on this forum; I now voted to close this as a duplicate of our generic SVD/PCA thread. Make sure you read that thread and navigate some linked threads as well. If after than some of your questions remain, edit your Q to clarify what exactly remains unclear. $\endgroup$ – amoeba Apr 25 '17 at 7:07
2
$\begingroup$

Here's the basic concept of PCA:

Let $x$ by some vector with some covariance matrix $\Sigma_x=E[(x-\mu_x)(x-\mu_x)^T]$. Next, recall the following theorem of random variable transformations: If $A$ is some matrix, then $y=Ax\Rightarrow \Sigma_y = A\Sigma_x A^T$.

Okay, so let's say you found some $A$ so that $A=\Sigma_x^{-0.5}$. Then, $$\Sigma_y = A\Sigma_x^{0.5}\Sigma_x^{0.5}A^T=I$$ where $I$ is the identity matrix. So by using this $A$ matrix, and multiplying $x$ by it, you got a decorrelated corrdinate system $y$ from your original vector, $x$.

How can you get this $A$? recall eigenvalue decomposition, $\Sigma_x u_i = \lambda_i u_i$, where $u_i,\lambda_i$ are eigenvector-eigenvalue pairs. In matrix form for all eigenvectors we have $$ \Sigma_x U = U \Lambda $$ where $\Lambda=diag(\lambda_1,\lambda_2,...)$. Now, since the covariance matrix is real and symmetric, we have the following property: $U^T=U^{-1}\Rightarrow U^TU=I$. So let's see what happens when multiplying the eigenvalue matrix equation by $U^T$:

$$ U^T \Sigma_x U = U^T U \Lambda = \Lambda. $$ Let's bring this equation together with the theorem from the beginning where $A=U^T$, so we have:

$$ y = U^T x \Rightarrow \Sigma_y = U^T\Sigma_x U = \Lambda, $$ where $\Lambda$ is a diagonal matrix. We can further scale by the inverse of eigenvalues (provided they're positive) and get a unit variance system.

To summarize, the eigenvector matrix of the covariance of $x$ was shown to be a transformation that decorrelates the random vector $x$.

$\endgroup$
  • $\begingroup$ hi, thanks a lot for the answer. i didn't understand this part: 'you got a decorrelated coordinate system y from your original vector, x'. how do we define decorrelated? also how is y a coordinate system by itself. thanks again. $\endgroup$ – Rafael Apr 25 '17 at 14:50
  • 1
    $\begingroup$ Uncorrelated means $E[x_i x_j]=0$ for $j\neq i$ (assuming mean 0). This is a weak form of dependency that's valid only for second moments. If you have uncorr. variables then they might still have dependency in higher order, unless their probability is defined with up to second moments, and then uncorrelated <-> independent. This happens for Gaussian vectors. You can think of this transformation as some abstract rotation of the coordinate system. For example, if you have a linear system such as $x_2=x_1+n_{noise}$, the $y$ system will be rotated by 45 degrees so that $y_2=0$. $\endgroup$ – yoki Apr 25 '17 at 16:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.