# Understanding Singular Value Decomposition in the context of LSI

My question is generally on Singular Value Decomposition (SVD), and particularly on Latent Semantic Indexing (LSI).

Say, I have $A_{word \times document}$ that contains frequencies of 5 words for 7 documents.

A =  matrix(data=c(2,0,8,6,0,3,1,
1,6,0,1,7,0,1,
5,0,7,4,0,5,6,
7,0,8,5,0,8,5,
0,10,0,0,7,0,0), ncol=7, byrow=TRUE)
rownames(A) <- c('doctor','car','nurse','hospital','wheel')


I get the matrix factorization for $A$ by using SVD: $A = U \cdot D \cdot V^T$.

s = svd(A)
D = diag(s$d) # singular value matrix S = diag(s$d^0.5 ) # diag matrix with square roots of singular values.


In 1 and 2, it is stated that:

$WordSim = U \cdot S$ gives the word similarity matrix, where the rows of $WordSim$ represent different words.

WordSim = s$u %*% S$DocSim= S \cdot V^T$gives the document similarity matrix where the columns of$DocSim$represent different documents. DocSim = S %*% t(s$v)

Questions:

1. Algebraically, why are $WordSim$ and $DocSimS$ word/document similarity matrices? Is there an intuitive explanation?
2. Based on the R example given, can we make any intuitive word count / similarity observations by just looking at $WordSim$ and $DocSim$ (without using cosine similarity or correlation coefficient between rows / columns)?

• I know very little about LSI, but SVD of a matrix is at the core of linear dimensionality-reduction, mapping methods, such as Principal components, biplots, Correspondence analysis. The main "laws" of SVD is that $AV=UD$ = projection of rows of $A$ onto the principal axes; and $A'U=VD'$ = projection of columns of $A$ onto the principal axes. In a sense, it is "similarity" values between the points (rows or columns) and the principal axes. Whether it can be treated as similarity between the points themselves is dependent on the context, I think. – ttnphns Jul 16 '14 at 15:04
• Ah.. I see in wikipedia that LSI is just correspondence analysis (CA). That's better. CA is the biplot of a specially prepared data table. The aforementioned projections or coordinates - you use them to plot row and column points in the space of the principal axes. Closeness between the row-row, col-col, and row-col points relate their similarity. However, the layout on the plot is dependent on how you spread inertia (variance) over the row and the col points. – ttnphns Jul 16 '14 at 15:12
• @ttnphns. Thank you, can you give a reference on: "$AV=UD$ = projection of rows of A onto the principal axes; and $A ′ U=VD ′$ = projection of columns of A onto the principal axes"? I think that will clarify things for me. By principal axes, do you mean the eigen vectors corresponding to the top m singular values in $D$? I also came across: "For PCA,we need not compute the left singular vectors", but cannot wholly comprehend why this is the case. – Zhubarb Jul 16 '14 at 15:37
• Your question could be improved by editing it to correctly reflect what that document states. On p. 22 it defines $S$ as containing the square roots of $D$, "restricted" to the largest ones. Therefore neither $UD$ nor $DV^\prime$ are involved, nor do they have interpretations as "similarity matrices." The relevant matrices are instead $US$ and $SV^\prime$. They can be used to reconstruct an approximation of $A=UDV^\prime\approx U(S^2)V^\prime=(US)(SV^\prime).$ – whuber Jul 16 '14 at 15:58