Using the 'U' Matrix of SVD as Feature Reduction [duplicate]

Question

This is a follow-up to the question asked regarding SVD and dimensionality reduction (question).

In that question I asked how to use SVD for dimensionality reduction. Although not stated, the ultimate goal here is to use the reduced feature set and input them into a classification or regression algorithm.

I have learned that SVD is a technique used by prcomp in R, as the "v" matrix from a run of svd on a centered and scaled matrix is the same as the loadings (eigen vectors) from a PCA using the traditional eigen decomposition on a correlation matrix:

data(iris)

#these two match
eigen(cor(iris[,-5])) #eigen vectors
svd(scale(iris[,-5]))$v

This has helped with my understanding of the connection between SVD and PCA. However, I have two additional questions:

1) Why do the following differ in signs for the first PC? Is this OK?

svd(cor(iris[,-5]))$u
svd(scale(iris[,-5]))$v

2) To match the output of prcomp, one can multiply the scaled/centered original data by the 'v' matrix from SVD:

PCSCORE1<-scale(iris[,-5]) %*% SVD2$v[,1:2]  #PC scores from SVD
PCSCORE1[1:10,]  #PC scores from first 2 PC

#matches this
PCA<-prcomp(iris[,-5], center = TRUE, scale =TRUE) 
PCA$x[1:10,1:2]

but I have seen in multiple locations (e.g. question) and the rapidminer package (a machine learning tool written in JAVA) that just the 'u' matrix that results from running svd on the center/scaled input matrix X is used as the PC scores.

What is the connection of u to Xv and if u can be used, why does prcomp compute Xv? Mechanically u is Xvdiag(1/d) so the eigen vectors related to the largest eigen values are scaled down - why is this used?

Answer 1

My (very limited) understanding is that u (left singular vectors) measures aspects of the rows of your matrix, and v (right singular vectors) measures aspects of the columns. So u can be used to measure similarity between rows, and v can be used to measure similarity between columns. If you multiply u and v, you get back an approximation of the original data matrix (this is how recommender engines work).

I would think you could use u as well as x %*% v as features in a learning algorithm.

Answer 2

From what I know, say you have a data matrix $X$ and it is $20$ x $1000$. Here, we have $20,000$ variables to keep track of and store/use.

Let us say you perform svd on your data matrix, such that:

$$ X = USV^H $$

Then, $U$ is a $20$ x $20$ matrix, $S$ is a $20$ x $1000$ matrix, and $V$ is a $1000$ x $1000$ matrix.

Now let us say you want to represent your data matrix more compactly. You inspect the diagonal elecment of $S$, and you decide you want to keep only the first two singular values/vectors, because the first two singular values are very very large relative to all the rest.

So now your compact data matrix, call it $X_{new}$, is made of the first two columns of $U$, the first two column/rows of $S$, and the first two columns of $V$.

But do you need to keep track of $20,000$ variables any more? No. You can just keep track of the first two columns of $U$, (20*2 = 40 numbers) the first two singular values of $S$, (2 numbers), and the first two columns of $V$, (1000 * 2 = 2000 numbers).

So now the total number of variables you need to keep track of / use / store / input into a classifier is: 40 + 2 + 2000 = 2042. Before this, you have to feed your classifier 20,000 variables. Now, you can feed it just 2042.

This is how I understand DR to work for classification.