2
$\begingroup$

How do we numerically implement SVD? I confused the numerically implementations between PCA and SVD (who implements who). Since we know that

  1. PCA can be numerically implemented by NIPALS (Non-linear iterative partial least-squares).

  2. And theoretically, we needn't to decompose the covariance matrix $X^TX$ in PCA instead we compute the SVD of $X.$

Then do we implement SVD by implementing corresponding PCA by NIPALS? Or implement SVD by other numerical method, then implement PCA by implementing corresponding SVD?

$\endgroup$

1 Answer 1

0
$\begingroup$

PCA

Principal component analysis seeks to obtain a set of new variables (the principal components, or scores) which are linear combinations of the original variables, following the equation

$$T = XP$$

where $T\in\mathbb{R}^{n\times m}$ is the scores matrix (the new variables), $X\in\mathbb{R}^{n\times m}$ is the original data matrix and $P\in\mathbb{R}^{m\times m}$ is the loadings matrix. These new variables are computed in a way such that the first principal component has the largest possible variance, and the second principal component has the largest possible variance subject to the restriction of being orthogonal to the previous one, and so on. This way, PCA provides with a set of orthogonal variables maximizing the oroginal variance from $X$.

The problem of PCA can then be posed as an optimization problem:

$$p = argmax(var(Xp))\quad s.t. \quad\|p\|=1$$

Eigen problem

Now let me assume without loss of generality, that $X$ is centered, so the mean of any variable in $X$ is equal to $0$. Given $\Sigma=X^\prime X$ the matrix of covariances from $X$, it can be seen, using linear algebra and the spectral theorem, that the solution to the optimization problem posed above is the matrix of eigenvectors of $\Sigma$.

SVD

In linear algebra, SVD, or singular value decomposition, is a factorization technique that generalizes the eigenvalue decomposition. Given a matrix $X$, the SVD decomposition of this matrix will have the form

$$X=UDV^\prime$$

  • $U$ is the matrix of eigenvectors of $X^\prime X$
  • $V$ is the matrix of eigenvectors of $X X^\prime$
  • $D$ is a diagonal matrix containing the singular values, which are the square roots of the eigenvalues of $X^\prime X$

So as it happens, one can solve the PCA problem in different ways, including:

  • Using a numerical algorithm to compute the eigenvectors of $X^\prime X$
  • Using a numerical algorithm to compute the SVD of $X$

And it can be seen that the NIPALS algorithm for PCA is simply an adaptation of the power method, an algorithm for numerically finding the eigenvectors of $X^\prime X$.

$\endgroup$
1
  • $\begingroup$ Using a numerical algorithm to compute the SVD of X this is my core question. What's the numerical algorithm, is it just the NIPALS on $X?$ if yes, then actually, SVD is not an alternative numerical way to solve PCA. Only when there is a totally different numerical way to solve SVD, we can say SVD is alternative to solve PCA apart from NIPALS. $\endgroup$ Jul 5, 2021 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.