Why is non-centered SVD accepted in LSA

Question

In Latent Semantic Analysis (LSA) , we apply SVD to a term-document matrix $A$, then choose to ignore all but $k$ largest singular values.

The term-document matrix is not centered, or normalised, because we want to preserve sparseness of $A$. Are there conditions under which the lack of centering is not considered a problem, given that it is known the results of PCA are misleading if the feature scales are different? (lots of good discussion here, e.g. How does centering make a difference in PCA (for SVD and eigen decomposition)? , PCA on correlation or covariance?)

Specifically, if $A$ were centered, we would be able to quantify how much variation is being lost for each choice of $k$. Is it fair to say that, in LSA, $k$ is being chosen blindly?

Answer 1

The Eckart-Young-Mirsky theorem provides that the best rank-$k$ approximation to $A$ is SVD that retains the $k$ largest singular vectors and singular values. There are several versions of the theorem, for different definitions of "best," such as Frobenius or spectral norm. But they all have in common that the best low-rank approximation is given by SVD. See: What norm of the reconstruction error is minimized by the low-rank approximation matrix obtained with PCA?

As you note, the non-centered SVD result will be different from the PCA result. However, the Eckart-Young-Mirsky theorem does not require centering $A$, so we still know that our rank-$k$ approximation is optimal.

Alternatively, you could view PCA as applying SVD to a centered, rescaled $A$. This post develops the relationships between SVD, PCA and centering of $A$ in more detail.

• Relationship between SVD and PCA. How to use SVD to perform PCA?

In any case, the key observation is that, for uncentered data, SVD will give a different result than PCA. However, even though uncentered SVD is different from PCA, it is still "optimal" in the sense of the Eckart-Young-Mirsky theorem. In the particular context of sparse data such as found in NLP, this can be important to know! (As an aside, a matrix which has its columns rescaled but not centered is still sparse because the zeros are just multiplied by some number, yielding zero. This fact can be important when considering data transformations of sparse $A$.)

SVD does not have the same relationship that PCA has to the covariance of the columns of $A$, so the $k$ largest eigenvalues do not correspond to some fraction of total variance. However, we can get at a similar idea when $A$ is a real matrix by considering the $k$ largest singular values as a fraction of the sum of the all of the singular values. Singular values are always non-negative for real $A$.

Or you could consider alternative criteria, more closely tied to whatever analysis goals you have in mind for your project:

• Do you have tight engineering tolerances regarding memory or computation time? You'll need to pick $k$ small enough to satisfy those.
• Do you need to keep a small parameter count, perhaps to reduce model training time? You'll need to pick $k$ small enough to hit that target.
• Do you find that the model quality is hurt by choosing $k$ too large (signal is overcome by noise) or to small (insufficient data)? You'll need to tune $k$.