20
$\begingroup$

I know how to calculate PCA and SVD mathematically, and I know that both can be applied to Linear Least Squares regression.

The main advantage of SVD mathematically seems to be that it can be applied to non-square matrices.

Both focus on the decomposition of the $X^\top X$ matrix. Other than the advantage of SVD mentioned, are there any additional advantages or insights provided by using SVD over PCA?

I'm really looking for the intuition rather than any mathematical differences.

$\endgroup$
  • 2
    $\begingroup$ The question is unclear. First you mention OLS regression. It then dissapears. Next, advantage... SVD over PCA - svd and PCA cannot be compared as a mathematical operation and data analytical method. Can your question be something about ways to do PCA? Or what are you asking? $\endgroup$ – ttnphns Oct 23 '14 at 11:58
  • 1
    $\begingroup$ Sorry for being unclear. I have ridge type estimators that are one derived using PCA and the other using SVD. There are differencs in the way the models are set up i.e. it terms of the prior information that they use. But they are written by the same author. I am trying to understand the differences between them and was trying to figure out why he would use PCA vs SVD as the basis for his analysis. Perhaps it was arbitrary, but if I can understand the pros and cons it would help. So far it seems SVD is just a way to do PCA that tends to be more numerically stable. $\endgroup$ – Baz Oct 23 '14 at 12:06
  • $\begingroup$ That's fine but I just wondered if using SVD also produces any additional econometric insights/intuition on the problem. $\endgroup$ – Baz Oct 23 '14 at 12:07
  • 1
    $\begingroup$ If you want specific focus on econometrics, I think you need to spell out that in the question and explain why. I can't see that a discussion of SVD and PCA, which are quite different kinds of beasts anyway, is different for econometrics than for any other branch of statistical science. $\endgroup$ – Nick Cox Oct 23 '14 at 12:43
  • 4
    $\begingroup$ @Baz: "So far it seems SVD is just a way to do PCA that tends to be more numerically stable" -- [in this context] it is exactly right, yes. $\endgroup$ – amoeba Oct 23 '14 at 16:00
41
$\begingroup$

As @ttnphns and @nick-cox said, SVD is a numerical method and PCA is an analysis approach (like least squares). You can do PCA using SVD, or you can do PCA doing the eigen-decomposition of $X^T X$ (or $X X^T$), or you can do PCA using many other methods, just like you can solve least squares with a dozen different algorithms like Newton's method or gradient descent or SVD etc.

So there is no "advantage" to SVD over PCA because it's like asking whether Newton's method is better than least squares: the two aren't comparable.

$\endgroup$
  • 8
    $\begingroup$ Nice example of how a concise, short answer can still get to the heart of a question. $\endgroup$ – Nick Cox Oct 23 '14 at 15:33
  • 3
    $\begingroup$ Wow, 8 upvotes for this answer and 0 upvotes for the original question. This does not make much sense. If you upvote the answer, consider upvoting the question as well! $\endgroup$ – amoeba Oct 23 '14 at 19:59
  • 1
    $\begingroup$ @amoeba The question to me is confused. The answer makes clear what the confusion is. I think that is a good explanation for differences in votes. $\endgroup$ – Nick Cox Oct 23 '14 at 23:50
  • 5
    $\begingroup$ Actually to be more pedantic, SVD is not a numerical method per se, it's a linear algebra operation, which can be implemented using specific numerical methods involving things like Householder transformations... $\endgroup$ – purple51 Oct 24 '14 at 0:05
  • $\begingroup$ Yet the advantage of (when deriving Principal Components via) SVD is a numerical one: more precision. See for example Jolliffe (2002). Maybe the $\endgroup$ – Nikos Alexandris May 13 '17 at 21:34
2
$\begingroup$

The question is really asking if you should do Z-score normalization of the columns before applying the SVD. This is because PCA is the above transformation followed by the SVD. Sometimes doing the normalization is quite harmful. If your data is for example (transformed) word counts which are positive, subtracting the mean is definitely harmful. This is because zeros which represent the absence of a word in a document will be mapped to negative numbers with high magnitude. In linear problems the higher magnitude should be used to represent the range where your features are most sensitive. Also dividing by the standard deviation is harmful for this type of data.

$\endgroup$
  • $\begingroup$ This is an interesting example, but I believe it should rather belong to some other thread. PCA can definitely be done without z-scoring, so I disagree with your first sentence: that's not what this question is "really asking". $\endgroup$ – amoeba Jul 20 '15 at 14:10
  • $\begingroup$ PCA and SVD are the same if you ignore subtracting the means (this is the Z-scoring I mentioned, sometimes people give the PCA with dividing by the stdev). So I disagree that you can do PCA without subtracting the means. You can do PCA on non-square matrices as well. $\endgroup$ – Stefan Savev Jul 20 '15 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.