Usually LSI/LSA is done on a TermFrequency matrix (each row a document, each column a term) to reduce the dimensionality along the terms dimension (i.e. there are too many words). In that way we would expect that similar terms are going to be put together and we obtain a cleaner / more readable representation of the documents.

My questions is: what if we apply the same technique on the document axis? Was this already studied? Does is makes sense at all?

The obtained matrix will have the same amount of columns but fewer rows. I would expect each row to represent a cluster of similar documents, but there are two things that are not too clear:

  • what is the ordering telling us? The first row corresponds to the highest eigenvalue, but I am not sure how to interpret that.

  • is there any interpretation of the coefficients in each row? Are those the terms that appear more frequently / have higher weight in the "cluster"?


L in LSI as you know stands for latent. So, LSI finds latent concepts in a given document term matrix (or matrix in general). In essence, LSI uses SVD. SVD is a matrix decomposition. If your input matrix is $A$ then the SVD of

$$A = U\Sigma V^T.$$

Here $A$ is $m \times n$.

$U$ is $m \times r$, indicating $m$ documents and $r$ concepts. This is a document to concept similarity matrix.

$\Sigma$ is $r \times r$. This is a diagonal matrix whose entries from upper left to lower right are positive and in decreasing order. These values are the eigenvalues of $A$. A higher value indicates the strength of the concept.

$V$ is $n \times r$ (note the transpose makes it $r \times n$), indicating $n$ terms and $r$ concepts. This is a term to concept similarity matrix.

So, SVD provides both concept strength in terms of documents ($U$), but also terms ($V$).

Like PCA, SVD can be used for dimensionality reduction. To do this you would reduce the square matrix $\Sigma$ of size $r$ to $\tilde{r}$ where $\tilde{r} < r$; retaining the top eigienvalues. (Also reduce dimensions $U$ and $V$ likewise.) This would produce a matrix $A'$ of lower dimension most representative of $A$.

So, to summarize. The eigenvalue ordering indicates (latent) concept strength. $U$ and $V$ are similarity matrices of document to concept and term to concept respectively.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.