Numerically PCA implements SVD or SVD implements PCA How do we numerically implement SVD? I confused the numerically implementations between PCA and SVD (who implements who). Since we know that

*

*PCA can be numerically implemented by NIPALS (Non-linear iterative partial least-squares).


*And theoretically, we needn't to decompose the covariance matrix $X^TX$ in PCA instead we compute the SVD of $X.$
Then do we implement SVD by implementing corresponding PCA by NIPALS? Or implement SVD by other numerical method, then implement PCA by implementing corresponding SVD?
 A: PCA
Principal component analysis seeks to obtain a set of new variables (the principal components, or scores) which are linear combinations of the original variables, following the equation
$$T = XP$$
where $T\in\mathbb{R}^{n\times m}$ is the scores matrix (the new variables), $X\in\mathbb{R}^{n\times m}$ is the original data matrix and $P\in\mathbb{R}^{m\times m}$ is the loadings matrix. These new variables are computed in a way such that the first principal component has the largest possible variance, and the second principal component has the largest possible variance subject to the restriction of being orthogonal to the previous one, and so on. This way, PCA provides with a set of orthogonal variables maximizing the oroginal variance from $X$.
The problem of PCA can then be posed as an optimization problem:
$$p = argmax(var(Xp))\quad s.t. \quad\|p\|=1$$
Eigen problem
Now let me assume without loss of generality, that $X$ is centered, so the mean of any variable in $X$ is equal to $0$. Given $\Sigma=X^\prime X$ the matrix of covariances from $X$, it can be seen, using linear algebra and the spectral theorem, that the solution to the optimization problem posed above is the matrix of eigenvectors of $\Sigma$.
SVD
In linear algebra, SVD, or singular value decomposition, is a factorization technique that generalizes the eigenvalue decomposition. Given a matrix $X$, the SVD decomposition of this matrix will have the form
$$X=UDV^\prime$$

*

*$U$ is the matrix of eigenvectors of $X^\prime X$

*$V$ is the matrix of eigenvectors of $X X^\prime$

*$D$ is a diagonal matrix containing the singular values, which are the square roots of the eigenvalues of $X^\prime X$
So as it happens, one can solve the PCA problem in different ways, including:

*

*Using a numerical algorithm to compute the eigenvectors of $X^\prime X$

*Using a numerical algorithm to compute the SVD of $X$
And it can be seen that the NIPALS algorithm for PCA is simply an adaptation of the power method, an algorithm for numerically finding the eigenvectors of $X^\prime X$.
