I've came across several great questions and answers here regarding PCA, but I would like to have a look at it from a linear transformation perspective. Let's say I have a (demeaned) data matrix $X$ which is $k\times n$, where $k$ is a number of variables and $n$ is a number of samples. We can estimate the covariance matrix by $S = XX' / (n-1)$. If we want to perform the PCA, what we would do next is to compute eigenvectors of $S$, sort them accroding to the eigenvalues in decreasing order and put them into the $k\times k$ matrix $V$. Finally, if I want to project my initial data, $X$, onto this new basis, I would do $V^{-1}X$ which is the same as $V'X$ since $V$ is orthonormal. But what I miss in this picture is why such a projection actually works the way it does. If I would do $S^{-1}X$ I would project onto a basis defined by the covariance matrix but this does not look like a useful procedure. I also know that the eigenvectors of $S$ are the ones that are unaffected by a transformation $S$ (not $S^{-1}$) up to a scaling factor and sign. I would really want to understand this connection between $V^{-1}$ and $S$ and how it works. I am familiar with other derivations of PCA, like e.g. solving a series of constrained minimisation problems, but interested of a basis transformation interpretation.

  • $\begingroup$ Re "does not look like a useful procedure:" $S^{-1}X$ maps any response $n$-vector onto its least-squares regression coefficients. BTW, one of the better ways to understand PCA is via SVD: see stats.stackexchange.com/questions/134282 for instance. $\endgroup$ – whuber May 21 at 15:08
  • $\begingroup$ "If I would do $S^{−1}X$ I would project onto a basis defined by the covariance matrix" - This is not true because $S^{-1}$ is not a matrix having basis vectors as its columns. $\endgroup$ – kasa May 21 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.