# Understanding Singular Value Decomposition in the context of LSI

My question is generally on Singular Value Decomposition (SVD), and particularly on Latent Semantic Indexing (LSI).

Say, I have $A_{word \times document}$ that contains frequencies of 5 words for 7 documents.

A =  matrix(data=c(2,0,8,6,0,3,1,
1,6,0,1,7,0,1,
5,0,7,4,0,5,6,
7,0,8,5,0,8,5,
0,10,0,0,7,0,0), ncol=7, byrow=TRUE)
rownames(A) <- c('doctor','car','nurse','hospital','wheel')


I get the matrix factorization for $A$ by using SVD: $A = U \cdot D \cdot V^T$.

s = svd(A)
D = diag(s$d) # singular value matrix S = diag(s$d^0.5 ) # diag matrix with square roots of singular values.


In 1 and 2, it is stated that:

$WordSim = U \cdot S$ gives the word similarity matrix, where the rows of $WordSim$ represent different words.

WordSim = s$u %*% S$DocSim= S \cdot V^T$gives the document similarity matrix where the columns of$DocSim$represent different documents. DocSim = S %*% t(s$v)

Questions:

1. Algebraically, why are $WordSim$ and $DocSimS$ word/document similarity matrices? Is there an intuitive explanation?
2. Based on the R example given, can we make any intuitive word count / similarity observations by just looking at $WordSim$ and $DocSim$ (without using cosine similarity or correlation coefficient between rows / columns)? • I know very little about LSI, but SVD of a matrix is at the core of linear dimensionality-reduction, mapping methods, such as Principal components, biplots, Correspondence analysis. The main "laws" of SVD is that $AV=UD$ = projection of rows of $A$ onto the principal axes; and $A'U=VD'$ = projection of columns of $A$ onto the principal axes. In a sense, it is "similarity" values between the points (rows or columns) and the principal axes. Whether it can be treated as similarity between the points themselves is dependent on the context, I think. Jul 16, 2014 at 15:04
• Ah.. I see in wikipedia that LSI is just correspondence analysis (CA). That's better. CA is the biplot of a specially prepared data table. The aforementioned projections or coordinates - you use them to plot row and column points in the space of the principal axes. Closeness between the row-row, col-col, and row-col points relate their similarity. However, the layout on the plot is dependent on how you spread inertia (variance) over the row and the col points. Jul 16, 2014 at 15:12
• @ttnphns. Thank you, can you give a reference on: "$AV=UD$ = projection of rows of A onto the principal axes; and $A ′ U=VD ′$ = projection of columns of A onto the principal axes"? I think that will clarify things for me. By principal axes, do you mean the eigen vectors corresponding to the top m singular values in $D$? I also came across: "For PCA,we need not compute the left singular vectors", but cannot wholly comprehend why this is the case. Jul 16, 2014 at 15:37
• Your question could be improved by editing it to correctly reflect what that document states. On p. 22 it defines $S$ as containing the square roots of $D$, "restricted" to the largest ones. Therefore neither $UD$ nor $DV^\prime$ are involved, nor do they have interpretations as "similarity matrices." The relevant matrices are instead $US$ and $SV^\prime$. They can be used to reconstruct an approximation of $A=UDV^\prime\approx U(S^2)V^\prime=(US)(SV^\prime).$
– whuber
Jul 16, 2014 at 15:58