4
$\begingroup$

In PCA it's common to center the data, i.e. preprocess the data matrix such that the columns have zero mean. PCA can be done via SVD, but in this case the data matrix also has to be mean-centered. If we don't center it, the found principal directions will not make sense.

But in LSA/LSI when we apply SVD to the term-document matrix, we do not center it - at least I haven't found any paper saying that we should.

For example, here's a figure from (1):

enter image description here

Although this illustration serves quite a different purpose (to show that NMF is better than LSI), we see data is not centered here. Why is that? Why we do not need centering for LSI?

The closest explanation that I found in (2) is that centering does not preserve the cosine similarity between documents.

But without centering the principal directions of LSI seem to be meaningless: in PCA we want to find directions of most variance, and if we don't center the data, the found directions will correspond to something else. So without centering it will be some projection, but not necessarily a good one in the PCA sense.

So why do we do LSI this way?

References:

  1. Xu, Wei, Xin Liu, and Yihong Gong. "Document clustering based on non-negative matrix factorization." 2003.
  2. Korenius, Tuomo, Jorma Laurikkala, and Martti Juhola. "On principal component analysis, cosine and Euclidean measures in information retrieval." 2007.
Improve this question
$\endgroup$
  • $\begingroup$ As wikipedia says, LSI is an application of correspondence analysis (which is, together with PCA is a specific case of svd-based biplot). Centering itself is not a core part of the analyses of this type, it is rather a pre-processing step. An important one, because results sharply depend on it. So, if there is sence or meaning to center - do center, if no - do not. I think that you should consult the LSI documentstion. $\endgroup$ – ttnphns May 18 '15 at 20:32
  • $\begingroup$ that centering does not preserve the cosine similarity between documents True. You'll get correlation coefficient then. without centering the principal directions of LSI are meaningless Why necessarily so? don't be hurry to say it. $\endgroup$ – ttnphns May 18 '15 at 21:00
  • $\begingroup$ I added a couple of lines why I think so. Maybe the problem comes from the fact that to me it looks like PCA and LSI are doing the same thing, but it's not necessarily true... Also, what LSI documentation you refer to? The seminal LSI paper doesn't seem to have an answer for that. $\endgroup$ – Alexey Grigorev May 19 '15 at 10:04
  • 1
    $\begingroup$ directions will correspond to something else Directions of maximal sum-of-squares. PCA or similar SVD-based methods always maximizes the explained SS. If the data cloud is centered then SS is called scatter and SS/n [or n-1] is called variance. See also. $\endgroup$ – ttnphns May 19 '15 at 14:40
  • 1
    $\begingroup$ I would agree with your three points. To combine, cosine similarity (Ochiai coefficient) looks more natural for document terms analysis than Pearson correlation. Primarily because cosine retains its relation with count nature of the task while correlation does not. $\endgroup$ – ttnphns May 24 '15 at 15:14
7
$\begingroup$

I believe there are following reasons for not centering the data for LSI:

  • Term-document matrices are typically very sparse, but if we center it, it will no longer be sparse and will be slower to process and occupy more space
  • Angle between documents is not preserved when centering the data
  • Here we don't care much about the variance: $k$-rank approximation $A \approx A_k = U_k \Sigma_k V_k^T$ (that reveals the "latent topics") minimizes the Frobenius norm of the reconstruction $\| A - A_k \|_F^2$ and it doesn't need centering for that.

Update:

Recently I came across another reason for not mean-centering data for term-document (or document-term) matrices when doing LSA/PCA:

Due to sparsity and high dimensionality of the data the mean was already close to 0, and we chose not to remove mean from the data.

It's footnote 1 from a paper by Leonid Zhukov and David Gleich "Topic identification in soft clustering using PCA and ICA". (pdf)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.