5
$\begingroup$

In PCA it's common to center the data, i.e. preprocess the data matrix such that the columns have zero mean. PCA can be done via SVD, but in this case the data matrix also has to be mean-centered. If we don't center it, the found principal directions will not make sense.

But in LSA/LSI when we apply SVD to the term-document matrix, we do not center it - at least I haven't found any paper saying that we should.

For example, here's a figure from (1):

enter image description here

Although this illustration serves quite a different purpose (to show that NMF is better than LSI), we see data is not centered here. Why is that? Why we do not need centering for LSI?

The closest explanation that I found in (2) is that centering does not preserve the cosine similarity between documents.

But without centering the principal directions of LSI seem to be meaningless: in PCA we want to find directions of most variance, and if we don't center the data, the found directions will correspond to something else. So without centering it will be some projection, but not necessarily a good one in the PCA sense.

So why do we do LSI this way?

References:

  1. Xu, Wei, Xin Liu, and Yihong Gong. "Document clustering based on non-negative matrix factorization." 2003.
  2. Korenius, Tuomo, Jorma Laurikkala, and Martti Juhola. "On principal component analysis, cosine and Euclidean measures in information retrieval." 2007.
$\endgroup$
6
  • $\begingroup$ As wikipedia says, LSI is an application of correspondence analysis (which is, together with PCA is a specific case of svd-based biplot). Centering itself is not a core part of the analyses of this type, it is rather a pre-processing step. An important one, because results sharply depend on it. So, if there is sence or meaning to center - do center, if no - do not. I think that you should consult the LSI documentstion. $\endgroup$
    – ttnphns
    Commented May 18, 2015 at 20:32
  • $\begingroup$ that centering does not preserve the cosine similarity between documents True. You'll get correlation coefficient then. without centering the principal directions of LSI are meaningless Why necessarily so? don't be hurry to say it. $\endgroup$
    – ttnphns
    Commented May 18, 2015 at 21:00
  • $\begingroup$ I added a couple of lines why I think so. Maybe the problem comes from the fact that to me it looks like PCA and LSI are doing the same thing, but it's not necessarily true... Also, what LSI documentation you refer to? The seminal LSI paper doesn't seem to have an answer for that. $\endgroup$ Commented May 19, 2015 at 10:04
  • 1
    $\begingroup$ directions will correspond to something else Directions of maximal sum-of-squares. PCA or similar SVD-based methods always maximizes the explained SS. If the data cloud is centered then SS is called scatter and SS/n [or n-1] is called variance. See also. $\endgroup$
    – ttnphns
    Commented May 19, 2015 at 14:40
  • 1
    $\begingroup$ I would agree with your three points. To combine, cosine similarity (Ochiai coefficient) looks more natural for document terms analysis than Pearson correlation. Primarily because cosine retains its relation with count nature of the task while correlation does not. $\endgroup$
    – ttnphns
    Commented May 24, 2015 at 15:14

1 Answer 1

9
$\begingroup$

I believe there are following reasons for not centering the data for LSI:

  • Term-document matrices are typically very sparse, but if we center it, it will no longer be sparse and will be slower to process and occupy more space
  • Angle between documents is not preserved when centering the data
  • Here we don't care much about the variance: $k$-rank approximation $A \approx A_k = U_k \Sigma_k V_k^T$ (that reveals the "latent topics") minimizes the Frobenius norm of the reconstruction $\| A - A_k \|_F^2$ and it doesn't need centering for that.

Update:

Recently I came across another reason for not mean-centering data for term-document (or document-term) matrices when doing LSA/PCA:

Due to sparsity and high dimensionality of the data the mean was already close to 0, and we chose not to remove mean from the data.

It's footnote 1 from a paper by Leonid Zhukov and David Gleich "Topic identification in soft clustering using PCA and ICA". (pdf)

$\endgroup$
3
  • $\begingroup$ Not sure I understand your third point. Distance minimization holds whether we center the matrix or not. But variance is precisely what we care about, and we want to measure it relative to the mean, not the origin... $\endgroup$ Commented Dec 12, 2021 at 18:39
  • $\begingroup$ In practice, not centering the matrix will put the most frequent terms into the dominant principal components, which is not desired. I think the true issue everyone's avoiding is computational intensity. I have a dataset of 1e6 documents and 5e4 terms with 5e7 nonzero entries. Truncated sparse SVD can handle that. In dense form you are dealing with 1000x more entries. $\endgroup$ Commented Dec 12, 2021 at 18:46
  • $\begingroup$ "Due to sparsity and high dimensionality of the data the mean was already close to 0, and we chose not to remove mean from the data." That doesn't sound right either. How can that be true for a matrix of nonnegative integers? $\endgroup$ Commented Dec 12, 2021 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.