28
$\begingroup$

I know how to calculate PCA and SVD mathematically, and I know that both can be applied to Linear Least Squares regression.

The main advantage of SVD mathematically seems to be that it can be applied to non-square matrices.

Both focus on the decomposition of the $X^\top X$ matrix. Other than the advantage of SVD mentioned, are there any additional advantages or insights provided by using SVD over PCA?

I'm really looking for the intuition rather than any mathematical differences.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ The question is unclear. First you mention OLS regression. It then dissapears. Next, advantage... SVD over PCA - svd and PCA cannot be compared as a mathematical operation and data analytical method. Can your question be something about ways to do PCA? Or what are you asking? $\endgroup$
    – ttnphns
    Oct 23, 2014 at 11:58
  • 1
    $\begingroup$ Sorry for being unclear. I have ridge type estimators that are one derived using PCA and the other using SVD. There are differencs in the way the models are set up i.e. it terms of the prior information that they use. But they are written by the same author. I am trying to understand the differences between them and was trying to figure out why he would use PCA vs SVD as the basis for his analysis. Perhaps it was arbitrary, but if I can understand the pros and cons it would help. So far it seems SVD is just a way to do PCA that tends to be more numerically stable. $\endgroup$
    – Baz
    Oct 23, 2014 at 12:06
  • $\begingroup$ That's fine but I just wondered if using SVD also produces any additional econometric insights/intuition on the problem. $\endgroup$
    – Baz
    Oct 23, 2014 at 12:07
  • 1
    $\begingroup$ If you want specific focus on econometrics, I think you need to spell out that in the question and explain why. I can't see that a discussion of SVD and PCA, which are quite different kinds of beasts anyway, is different for econometrics than for any other branch of statistical science. $\endgroup$
    – Nick Cox
    Oct 23, 2014 at 12:43
  • 4
    $\begingroup$ @Baz: "So far it seems SVD is just a way to do PCA that tends to be more numerically stable" -- [in this context] it is exactly right, yes. $\endgroup$
    – amoeba
    Oct 23, 2014 at 16:00

2 Answers 2

Reset to default
60
$\begingroup$

As @ttnphns and @nick-cox said, SVD is a numerical method and PCA is an analysis approach (like least squares). You can do PCA using SVD, or you can do PCA doing the eigen-decomposition of $X^T X$ (or $X X^T$), or you can do PCA using many other methods, just like you can solve least squares with a dozen different algorithms like Newton's method or gradient descent or SVD etc.

So there is no "advantage" to SVD over PCA because it's like asking whether Newton's method is better than least squares: the two aren't comparable.

Improve this answer
$\endgroup$
4
  • 9
    $\begingroup$ Nice example of how a concise, short answer can still get to the heart of a question. $\endgroup$
    – Nick Cox
    Oct 23, 2014 at 15:33
  • 1
    $\begingroup$ @amoeba The question to me is confused. The answer makes clear what the confusion is. I think that is a good explanation for differences in votes. $\endgroup$
    – Nick Cox
    Oct 23, 2014 at 23:50
  • 6
    $\begingroup$ Actually to be more pedantic, SVD is not a numerical method per se, it's a linear algebra operation, which can be implemented using specific numerical methods involving things like Householder transformations... $\endgroup$
    – purple51
    Oct 24, 2014 at 0:05
  • $\begingroup$ Yet the advantage of (when deriving Principal Components via) SVD is a numerical one: more precision. See for example Jolliffe (2002). Maybe the $\endgroup$
    – Nikos Alexandris
    May 13, 2017 at 21:34
5
$\begingroup$

The question is really asking if you should do Z-score normalization of the columns before applying the SVD. This is because PCA is the above transformation followed by the SVD. Sometimes doing the normalization is quite harmful. If your data is for example (transformed) word counts which are positive, subtracting the mean is definitely harmful. This is because zeros which represent the absence of a word in a document will be mapped to negative numbers with high magnitude. In linear problems the higher magnitude should be used to represent the range where your features are most sensitive. Also dividing by the standard deviation is harmful for this type of data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is an interesting example, but I believe it should rather belong to some other thread. PCA can definitely be done without z-scoring, so I disagree with your first sentence: that's not what this question is "really asking". $\endgroup$
    – amoeba
    Jul 20, 2015 at 14:10
  • $\begingroup$ PCA and SVD are the same if you ignore subtracting the means (this is the Z-scoring I mentioned, sometimes people give the PCA with dividing by the stdev). So I disagree that you can do PCA without subtracting the means. You can do PCA on non-square matrices as well. $\endgroup$
    – Stefan Savev
    Jul 20, 2015 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.