These terms get thrown around together a lot, but I would like to know what you think the differences are, if any.



3 Answers 3


LSA and LSI are mostly used synonymously, with the information retrieval community usually referring to it as LSI. LSA/LSI uses SVD to decompose the term-document matrix A into a term-concept matrix U, a singular value matrix S, and a concept-document matrix V in the form: A = USV'. The wikipedia page has a detailed description of latent semantic indexing.


Notably while LSA and LSI use SVD to do their magic, there is a computationally and conceptually simpler method called HAL (Hyperspace Analogue to Language) that sifts through text keeping track of preceding and subsequent contexts. Vectors are extracted from these (often weighted) co-occurrence matrices and specific words are selected to index the semantic space. In many ways I'm given to understand it performs as well as LSA without requiring the mathematically/conceptually complex step of SVD. See Lund & Burgess, 1996 for details.

  • 4
    $\begingroup$ ...recapitulating prior work by Finch and Chater (1992, 1994), Schütze (1993), and others. HAL, LSA, and other prior art work by generating a similarity measure for words by computing their contextual similarity. (This is Shephard's 'second order' similarity: 'first order' similarity is when word a occurs near word b; 'second order' similarity is that word a occurs near the same sorts of words as word b does). $\endgroup$ Dec 23, 2010 at 23:07
  • 3
    $\begingroup$ Comparing and contrasting: For LSA the context is the complete document. For HAL and others it is a textual window surrounding the target word. LSA measures distance in a linear subspace extracted via SVD/PCA, and the other deal with distances in the original space of surrounding word counts. $\endgroup$ Dec 23, 2010 at 23:09

NMF and SVD are both matrix factorization algorithms. Wikipedia has some relevant information on NMF.

SVD and PCA are intimately related. For starters, PCA is simply the eigendecomposition of the correlation. SVD is a generalization of eigendecomposition to non-square matrices. The singular values are the square root of the eigenvalues of the matrix multiplied by its transpose (making it square, and amenable to eigendecomposition). Furthermore, if the matrix is normal ($A^*A=A A^*$), the singular values are simply the absolute values of the eigenvalues. In any case, the singular values are non-negative, and losing the sign of the eigenvalues is the price you pay for being able to work with non-square matrices.

The other responders have covered LSI/LSA...

  • $\begingroup$ it should be covariance matrix, right? not the correlation matrix. $\endgroup$
    – Rafael
    Mar 30, 2017 at 5:34
  • $\begingroup$ Yes, unless you center your variables first. $\endgroup$
    – Emre
    Mar 30, 2017 at 6:05
  • $\begingroup$ after normalization of variables, it becomes correlation matrix? $\endgroup$
    – Rafael
    Mar 30, 2017 at 6:31
  • $\begingroup$ Normalization is centering with scaling, so that's different. $\endgroup$
    – Emre
    Mar 30, 2017 at 6:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.