5
$\begingroup$

I understand that PCA and SVD are similar - PCA removes the mean and SVD doesn't? I think I have an understanding of PCA - you would use it to reduce dimensions of data and separate it out into linear combinations of variables that explain the largest variance of the SS.

But I can't grasp that same concept for SVD - and especially can't understand when you would use SVD over PCA if they are supposed to be very similar.

Can someone explain in very basic terms why I would pick SVD vs PCA, and how SVD works? What is a real world application where SVD is better than PCA?

Improve this question
$\endgroup$
2
  • 6
    $\begingroup$ Read stats.stackexchange.com/q/79043/3277. PCA and SVD are not comparable at all. PCA is a data analytic method. SVD is a mathematical operation. PCA is often done via SVD (BTW PCA does not necessarily remove means). Some other analytical methods, similar to PCA - Correspondence analysis, Principal Coordinate analysis, Procrustes rotation etc. - use SVD also. $\endgroup$
    – ttnphns
    Commented Aug 14, 2014 at 6:52
  • $\begingroup$ Related: stats.stackexchange.com/questions/202036. $\endgroup$
    – amoeba
    Commented Mar 16, 2016 at 20:43

1 Answer 1

Reset to default
6
$\begingroup$

From what I understand. SVD has to do with general matrix factorization in some sense. It can be called as a numerical method to do factorization such the we can decompose $A = UDV^T$ such that D is diagonal and U and V are orthogonal. Thats it.

Now if we formulate the problem of PCA i.e. find all directions of maximum variation and those directions should be orthogonal and try finding the solution. One solution comes out to be the directions being eigen vectors of the covariance matrix. Thats why SVD can be used to find the solution of PCA. The eigen vectors are the principal components and its importance is based on the eigen value of each eigen vector. (http://cs229.stanford.edu/notes/cs229-notes10.pdf see page 5 footnote of how it comes to be eigen vectors).

Improve this answer
$\endgroup$
2
  • $\begingroup$ The notes don't actually show how SVD singular values turned to principal components. $\endgroup$
    – SmallChess
    Commented Nov 16, 2015 at 4:10
  • 2
    $\begingroup$ look at page 5 footnote. Now it turns down to solving $Ax=\lambda x$ and this is an eigen value problem. There are many ways to solve $Ax=\lambda x$ and you can use any one of them. Solutions to this is called eigen vectors(for x) and eigen values(for $\lambda$). You should look into linear algebra for more perspectives. $\endgroup$
    – dksahuji
    Commented Nov 16, 2015 at 7:06

Not the answer you're looking for? Browse other questions tagged or ask your own question.