Latent Semantic Indexing vs. PCA

Question

I am trying to understand how Latent Semantic Analysis works, reading demonstrations based on singular value decomposition.

Let's denote $X$ a $N \times D$ document-term matrix. The $D$ rows of $X$ represent documents, and the $W$ columns represent words. Using SVD, we can write $X = U \Sigma V^T$.

To me, this formulation is nothing more than a non-centered PCA. So then $WordSim = U \cdot S$ gives the word similarity matrix, where the rows of $WordSim $ represent different words and $DocSim= S \cdot V^T$ gives the document similarity matrix where the columns of $DocSim$ represent different documents. This would be akin to projecting the rows or columns onto the principal components of the $X^{T}X$ or $XX^{T}$, which is just like PCA.

Assume we have a new document $d$. We would like to get a rank $k$ approximation of $d$ by applying the same mapping that transformed rows in $X$ into rows in $\tilde{X}$ and get $\hat{d}$. Wikipedia has it that $\hat{d} = \Sigma^{-1}_k U_k^T d$, but I can't find a full, formal mathematical demonstration behind this equation. I don't understand why we can't just directly project $d$ onto $U^{T}$, as suggested here?

Answer 1

Preamble: We often do a TFIDF transform before feeding that matrix to the SVD. In addition: $U$ is term matrix (holding left singular vectors) and $V$ is the (potentially truncated) document matrix (holding right singular vectors), both are orthonormal.

To your main question: In the example shown in Wikipedia for a document approximation ($\hat{d} = \Sigma^{-1}_k U^T_kd$), we multiply by the inverse of $\Sigma$ to whiten our data when projecting them to the new document-space; the elements in the diagonal matrix $\Sigma$ can be though as "standard deviations" so by multiplying with the inverse of them we ensure that the projected scores will have unit variance. Note that with LSA, contrary to standard PCA, the input data in LSA example are not usually preprocessed in any way; nevertheless by construction the components in $U$ still define an orthonormal basis (for $XX^T$). Examining now to the Medium.com article linked: Our (query) document approximation is given as $\hat{d} = U^Td$. $\Sigma V^T$ is the projection of our 9 documents in the rescaled document-space. We note that in this document-space the projections are rescaled by $\Sigma$. Therefore when projecting a new document $d_i$ in that space (the one from $\Sigma V^T$) it makes sense to use $U^T$ only because we want to have projections in their original scale, we don't want to whiten them. As I see it, the author tried (and succeeded) to match gensim output/conventions. Both sources are correct, just they differ on where they expect $\hat{d}$ to reside.