3
$\begingroup$

This question already has an answer here:

This is a follow-up to the question asked regarding SVD and dimensionality reduction (question).

In that question I asked how to use SVD for dimensionality reduction. Although not stated, the ultimate goal here is to use the reduced feature set and input them into a classification or regression algorithm.

I have learned that SVD is a technique used by prcomp in R, as the "v" matrix from a run of svd on a centered and scaled matrix is the same as the loadings (eigen vectors) from a PCA using the traditional eigen decomposition on a correlation matrix:

data(iris)

#these two match
eigen(cor(iris[,-5])) #eigen vectors
svd(scale(iris[,-5]))$v

This has helped with my understanding of the connection between SVD and PCA. However, I have two additional questions:

1) Why do the following differ in signs for the first PC? Is this OK?

svd(cor(iris[,-5]))$u
svd(scale(iris[,-5]))$v

2) To match the output of prcomp, one can multiply the scaled/centered original data by the 'v' matrix from SVD:

PCSCORE1<-scale(iris[,-5]) %*% SVD2$v[,1:2]  #PC scores from SVD
PCSCORE1[1:10,]  #PC scores from first 2 PC

#matches this
PCA<-prcomp(iris[,-5], center = TRUE, scale =TRUE) 
PCA$x[1:10,1:2]

but I have seen in multiple locations (e.g. question) and the rapidminer package (a machine learning tool written in JAVA) that just the 'u' matrix that results from running svd on the center/scaled input matrix X is used as the PC scores.

What is the connection of u to Xv and if u can be used, why does prcomp compute Xv? Mechanically u is Xvdiag(1/d) so the eigen vectors related to the largest eigen values are scaled down - why is this used?

$\endgroup$

marked as duplicate by amoeba, gung, Nick Cox, whuber Jan 22 '15 at 16:43

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Question (1) is answered in several places, including stats.stackexchange.com/questions/34396/…. $\endgroup$ – whuber Jul 17 '13 at 20:02
  • $\begingroup$ Best I have found in multiple statements (other questions) that the solutions are not unique and thus signs can flip. I guess that explains this - although I read those more as explaining differences in software. $\endgroup$ – B_Miner Jul 17 '13 at 20:28
  • 1
    $\begingroup$ It can help to think about this geometrically. Principal components are lines without direction. The software outputs vectors to describe those lines. Literally any nonzero vector on a line will suffice, but typically only vectors of unit length are used--of which there are, of course, two. Furthermore, in (rare) cases there are two or more principal lines with the same eigenvalue. They generate entire planes (or higher dimensional linear subspaces) of "principal components." This creates even more ways to represent the answer--you can rotate the vectors in these planes. $\endgroup$ – whuber Jul 17 '13 at 20:45
2
$\begingroup$

My (very limited) understanding is that u (left singular vectors) measures aspects of the rows of your matrix, and v (right singular vectors) measures aspects of the columns. So u can be used to measure similarity between rows, and v can be used to measure similarity between columns. If you multiply u and v, you get back an approximation of the original data matrix (this is how recommender engines work).

I would think you could use u as well as x %*% v as features in a learning algorithm.

$\endgroup$
  • $\begingroup$ I think your interpretation of u and v are correct - certainly aligns with my understanding of LSI. I am not completely clear on why scaling x %*% v by the reciprocal of d makes any intuitive sense? I guess perhaps it accentuates the PC scores that have smaller variance this way? $\endgroup$ – B_Miner Jul 17 '13 at 20:11
  • 1
    $\begingroup$ @B_Miner I've never seen a good theoretical reason to add the eigenvalues into the calculation, but I've seen it improve predictive models, so I try it sometimes. $\endgroup$ – Zach Jul 17 '13 at 20:40
0
$\begingroup$

From what I know, say you have a data matrix $X$ and it is $20$ x $1000$. Here, we have $20,000$ variables to keep track of and store/use.

Let us say you perform svd on your data matrix, such that:

$$ X = USV^H $$

Then, $U$ is a $20$ x $20$ matrix, $S$ is a $20$ x $1000$ matrix, and $V$ is a $1000$ x $1000$ matrix.

Now let us say you want to represent your data matrix more compactly. You inspect the diagonal elecment of $S$, and you decide you want to keep only the first two singular values/vectors, because the first two singular values are very very large relative to all the rest.

So now your compact data matrix, call it $X_{new}$, is made of the first two columns of $U$, the first two column/rows of $S$, and the first two columns of $V$.

But do you need to keep track of $20,000$ variables any more? No. You can just keep track of the first two columns of $U$, (20*2 = 40 numbers) the first two singular values of $S$, (2 numbers), and the first two columns of $V$, (1000 * 2 = 2000 numbers).

So now the total number of variables you need to keep track of / use / store / input into a classifier is: 40 + 2 + 2000 = 2042. Before this, you have to feed your classifier 20,000 variables. Now, you can feed it just 2042.

This is how I understand DR to work for classification.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.