I am looking to explore using LSA as a dimension reduction technique for some textual data. To be specific I would like to take a matrix of M documents and N n-grams (ie: variables) and then create a smaller set of D variables that accounts for the most of the variance (thus resulting in a MxD matrix). The problem I am having is that as far as I can see lsa's fold_in() function only seems to return a full MxN matrix (albeit one that was estimated using D number of singular values).

Here is an example of what I mean:

> library(lsa)
> #make training set
> td = tempfile()
> dir.create(td)
> write( c("dog", "cat", "mouse"), file=paste(td, "D1", sep="/") )
> write( c("ham", "mouse", "sushi"), file=paste(td, "D2", sep="/") )
> write( c("dog", "pet", "pet"), file=paste(td, "D3", sep="/") )
> write( c("dog", "pet", "bird","cat"), file=paste(td, "D4", sep="/") )
> train = textmatrix(td)
> #make test set
> td = tempfile()
> dir.create(td)
> write( c("dog", "mouse","bird"), file=paste(td, "D1", sep="/") )
> write( c("ham", "mouse", "sushi"), file=paste(td, "D2", sep="/") )
> write( c("dog", "pet", "pet","cat"), file=paste(td, "D3", sep="/") )
> test = textmatrix(td)
> #run LSA withing training data, reducing dimension to 2
> train.lsa = lsa(train, dims=2)
> #apply LSA to test data
> (test.lsa <- t(fold_in(test, train.lsa)))

docs       cat       dog      mouse         ham       sushi        pet       bird
  D1 0.6477117 0.7120160 0.85915463  0.43561856  0.43561856  0.3527842 0.22417560
  D2 0.4613586 0.2370914 1.03194173  0.59632318  0.59632318 -0.4227943 0.02574004
  D3 0.5074166 0.9633718 0.04236298 -0.08829291 -0.08829291  1.2886710 0.37676074

As you can see the resulting test.lda object is 3x7 whereas I am looking to reduce the number of number of variables to 2. I could apply prcomp() to as.textmatrix(train.lsa) and then predict.prcomp() to test.lsa to reduce dimension after the fact, but that seems redundant given that SVD is already being used to run the LSA and that only 2 singular values are being used to create test.lsa to being with. I assume that I could also, after reviewing a bit of linear algebra, use the SVD components stored within the train.lsa object to do some manual matrix multiplication to get what I'm looking for but the lack of an easy to use function in the lsa package to do this makes me weary that I'm missing something here. So my question is what am I missing? Is there a simple function that I just can't find or am I misunderstanding how LSA is supposed to be used for "dimension reduction"?

Also I apologize if this question is better suited for cross-validated - I'm kind of hoping this just a technical issue of me not seeing some function or parameter in the lsa package so I'm going with stackoverflow to start with.


migrated from stackoverflow.com Jul 9 '13 at 2:16

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ What's LSA? Presumably all of us are open to learning something new, so it's best not to assume that everyone knows your abbreviations. $\endgroup$ – Nick Cox Jul 10 '13 at 14:10
  • $\begingroup$ Latent Semantic Analysis $\endgroup$ – David Jul 10 '13 at 15:17

No bites so I'd thought I'd post + accept what I've come up until someone, hopefully,can do better.

I think that this new fold_in() function will do what I want:

fold_in2 <- function(train.lsa,test){
  return(crossprod(t(crossprod(test, train.lsa$tk)), solve(diag(train.lsa$sk))))

> fold_in2(train.lsa,test)

docs        [,1]       [,2]
  D1 -0.37404506  0.5325658
  D2 -0.06960317  0.7690764
  D3 -0.58860967 -0.1680743

My only hesitation with it is that I'm not 100% on the linear algebra behind it so I will likely just use straight up PCA via prcomp() - after looking at the actual code used for lsa() I'm not seeing anything special that makes it better than PCA to begin with.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.