LSA vs. PCA (document clustering)

Question

I'm investigation various techniques used in document clustering and I would like to clear some doubts concerning PCA (principal component analysis) and LSA (latent semantic analysis).

First thing - what are the differences between them? I know that in PCA, SVD decomposition is applied to term-covariance matrix, while in LSA it's term-document matrix. Is there anything else?

Second - what's their role in document clustering procedure? From what I have read so far, I deduce that their purpose is reduction of the dimensionality, noise reduction and incorporating relations between terms into the representation. After executing PCA or LSA, traditional algorithms like k-means or agglomerative methods are applied on the reduced term space and typical similarity measures, like cosine distance are used. Please correct me if I'm wrong.

Third - does it matter if the TF/IDF term vectors are normalized before applying PCA/LSA or not? And should they be normalized again after that?

Fourth - let's say I have performed some clustering on the term space reduced by LSA/PCA. Now, how should I assign labels to the result clusters? Since the dimensions don't correspond to actual words, it's rather a difficult issue. The only idea that comes to my mind is computing centroids for each cluster using original term vectors and selecting terms with top weights, but it doesn't sound very efficient. Are there some specific solutions for this problem? I wasn't able to find anything.

I will be very grateful for clarifying these issues.

Answer 1

1. PCA and LSA are both analyses which use SVD. PCA is a general class of analysis and could in principle be applied to enumerated text corpora in a variety of ways. In contrast LSA is a very clearly specified means of analyzing and reducing text. Both are leveraging the idea that meaning can be extracted from context. In LSA the context is provided in the numbers through a term-document matrix. In the PCA you proposed, context is provided in the numbers through providing a term covariance matrix (the details of the generation of which probably can tell you a lot more about the relationship between your PCA and LSA). You may want to look here for more details.
2. You are basically on track here. The exact reasons they are used will depend on the context and the aims of the person playing with the data.
3. The answer will probably depend on the implementation of the procedure you are using.
4. Carefully and with great art. Most consider the dimensions of these semantic models to be uninterpretable. Note that you almost certainly expect there to be more than one underlying dimension. When there is more than one dimension in factor analysis, we rotate the factor solution to yield interpretable factors. However, for some reason this is not typically done for these models. Your approach sounds like a principled way to start your art... although I'd be less than certain the scaling between dimensions is similar enough to trust a cluster analysis solution. If you want to play around with meaning, you might also consider a simpler approach in which the vectors have a direct relationship with specific words, e.g. HAL.

Answer 2

LSI is computed on the term-document matrix, while PCA is calculated on the covariance matrix, which means LSI tries to find best linear subspace to describe the data set, while PCA tries to find the best parallel linear subspace.

Answer 3

Just some extension to russellpierce's answer.

1) Essentially LSA is PCA applied to text data. When using SVD for PCA, it's not applied to the covariance matrix but the feature-sample matrix directly, which is just the term-document matrix in LSA. The difference is PCA often requires feature-wise normalization for the data while LSA doesn't.

There's a nice lecture by Andrew Ng that illustrates the connections between PCA and LSA.

2/3) Since document data are of various lengths, usually it's helpful to normalize the magnitude. Here sample-wise normalization should be used not the feature-wise normalization. In practice I found it helpful to normalize both before and after LSI.

If the clustering algorithm metric does not depend on magnitude (say cosine distance) then the last normalization step can be omitted.

4) It think this is in general a difficult problem to get meaningful labels from clusters. Some people extract terms/phrases that maximize the difference in distribution between the corpus and the cluster. Another way is to use semi-supervised clustering with predefined labels.