# Understanding PCA from a linear transformation perspective

I've came across several great questions and answers here regarding PCA, but I would like to have a look at it from a linear transformation perspective. Let's say I have a (demeaned) data matrix $$X$$ which is $$k\times n$$, where $$k$$ is a number of variables and $$n$$ is a number of samples. We can estimate the covariance matrix by $$S = XX' / (n-1)$$. If we want to perform the PCA, what we would do next is to compute eigenvectors of $$S$$, sort them accroding to the eigenvalues in decreasing order and put them into the $$k\times k$$ matrix $$V$$. Finally, if I want to project my initial data, $$X$$, onto this new basis, I would do $$V^{-1}X$$ which is the same as $$V'X$$ since $$V$$ is orthonormal. But what I miss in this picture is why such a projection actually works the way it does. If I would do $$S^{-1}X$$ I would project onto a basis defined by the covariance matrix but this does not look like a useful procedure. I also know that the eigenvectors of $$S$$ are the ones that are unaffected by a transformation $$S$$ (not $$S^{-1}$$) up to a scaling factor and sign. I would really want to understand this connection between $$V^{-1}$$ and $$S$$ and how it works. I am familiar with other derivations of PCA, like e.g. solving a series of constrained minimisation problems, but interested of a basis transformation interpretation.

• Re "does not look like a useful procedure:" $S^{-1}X$ maps any response $n$-vector onto its least-squares regression coefficients. BTW, one of the better ways to understand PCA is via SVD: see stats.stackexchange.com/questions/134282 for instance. – whuber May 21 at 15:08
• "If I would do $S^{−1}X$ I would project onto a basis defined by the covariance matrix" - This is not true because $S^{-1}$ is not a matrix having basis vectors as its columns. – kasa May 21 at 16:28