4
$\begingroup$

After reading several "tutorials" on SVD I am left still wondering how to use it for dimensionality reduction.

Here is my confusion in an applied setting. If I limit svd to only considering the first two singular values / vectors and "recreate" the matrix, the dimensionality is still the same (4 columns). What should be done here to instead only use 2 columns?

data(iris)
s<-svd(iris[,-5])

u<-as.matrix(s$u[,1:2])
    v<-as.matrix(s$v[,1:2])
d<-as.matrix(diag(sing$d)[1:2, 1:2])

s2<-u%*%d%*%t(v)
Improve this question
$\endgroup$
6
  • $\begingroup$ Marc in the box, where is the use of SVD discussed in this question? $\endgroup$
    – B_Miner
    Jul 15, 2013 at 14:26
  • $\begingroup$ I can't read R code, but it looks to me like you perform svd on a rectangular data matrix. If you aim to do PCA by means of svd, you should decompose covariance/correlation or other square inner product matrix. $\endgroup$
    – ttnphns
    Jul 15, 2013 at 14:28
  • 1
    $\begingroup$ @B_Miner It's kind of hidden, but SVD is one of the common methods of obtaining principal components. Since you're using R, have a look at ?prcomp. $\endgroup$
    – Hong Ooi
    Jul 15, 2013 at 14:29
  • $\begingroup$ Hong and ttnphns, maybe this is my lack of understanding but is SVD the same as PCA? I am familiar with PCA academically, but was under the impression SVD was a different technique. I am looking at things such as ling.ohio-state.edu/~kbaker/pubs/… $\endgroup$
    – B_Miner
    Jul 15, 2013 at 14:32
  • $\begingroup$ @B_Miner - Yes, sorry for the confusion. As @Hongooi points out, svd is the main algorithm behind PCA and thus the other post is asking a very similar question. By "dimension reduction", what is really meant is that a smaller number of linear predictors can be used to explain a large portion of the data. $\endgroup$
    – Marc in the box
    Jul 15, 2013 at 14:38

1 Answer 1

Reset to default
5
$\begingroup$

I think your confusion comes from the fact that the PCA truncation is going to reconstruct the full dimensions of the original matrix. If you want to only consider the first two columns of the data, then this has to be what you decompose with svd.

The first example is a truncation of the iris data using all 4 columns (as in your example) and then truncating with one PC:

dat <- as.matrix(iris[,-5])
s <- svd(dat)
plot(cumsum(s$d^2/sum(s$d^2))) # % explained variance

pc.use <- 1
recon <- s$u[,pc.use] %*% diag(s$d[pc.use], length(pc.use), length(pc.use)) %*% t(s$v[,pc.use])

x11(6,6)
par(mfcol=c(1,2), mar=c(1,1,1,1), oma=c(0,3,1,0))
zlim=range(dat, recon)
image(dat, zlim=zlim, yaxt="n", xaxt="n", ylab="", xlab="", main="Iris data")
axis(2, at=seq(0,1,,ncol(dat)), labels=colnames(dat))
image(recon, zlim=zlim, yaxt="n", xaxt="n", ylab="", xlab="", main="Truncated")

enter image description here

The second example is an svd on only the first two columns of iris, thus the reconstruction is also only going to have two columns. The reconstruction again uses the single leading PC:

dat <- as.matrix(iris[,-c(3:5)])
s <- svd(dat)
plot(cumsum(s$d^2/sum(s$d^2))) # % explained variance

pc.use <- 1
recon <- s$u[,pc.use] %*% diag(s$d[pc.use], length(pc.use), length(pc.use)) %*% t(s$v[,pc.use])

x11(6,6)
par(mfcol=c(1,2), mar=c(1,1,1,1), oma=c(0,3,1,0))
zlim=range(dat, recon)
image(dat, zlim=zlim, yaxt="n", xaxt="n", ylab="", xlab="", main="Iris data")
axis(2, at=seq(0,1,,ncol(dat)), labels=colnames(dat))
image(recon, zlim=zlim, yaxt="n", xaxt="n", ylab="", xlab="", main="Truncated")

enter image description here

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Marc, I am confused where the dimensionality reduction is occurring. In the first example, we still end up with 4 columns and while the second example results in 2 columns, it only used the first two columns of iris - which is a feature selection, not a dimensionality reduction. I expected SVD to give a similar result as a PCA, where the full data is used, but less than the original number of columns results. $\endgroup$
    – B_Miner
    Jul 15, 2013 at 15:51
  • 1
    $\begingroup$ The truncation occurs by only using a subset of the PCs (in the example, only the 1st one), but the reconstructed data has the same dimensions as the original. $\endgroup$
    – Marc in the box
    Jul 15, 2013 at 20:26
  • $\begingroup$ Marc, do you mean using s$u[,pc.use] as the "new" matrix (say, as input to a predictive algorithm)? $\endgroup$
    – B_Miner
    Jul 15, 2013 at 22:39
  • $\begingroup$ @B_Miner - I think you are mixing concepts here. By using a smaller number of PCs (e.g. s$u[,1]...) the reconstruction of the iris data is truncated. i.e. the reconstruction of the iris data contains a large part of the variance, but the reconstruction is perfect only if all PCs are used. Using the PCs for prediction usually requires projecting a new data set onto the PCs in order to return a new set of loadings: new.u <- new.dat %*% s$v %*% solve(diag(s$d)). This is essentially what predict() does with class : prcomp. $\endgroup$
    – Marc in the box
    Jul 18, 2013 at 6:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.