Natural Language Processing: Basic Dimension Reduction with SVD of a Co-Occurence Matrix

Given sentences

1. I enjoy flying.
2. I like NLP.
3. I like deep learning

We can form a Co-Occurrence Matrix as follows:

Now we can apply Singular Value Decomposition to this matrix to get $$X = U \Sigma V^T$$ where $$U$$ and $$V$$ are orthogonal and $$\Sigma$$ is diagonal and the singular values are sorted (I believe) so $$\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_r$$.

My question is what do the matrices $$U$$, $$\Sigma$$, $$V$$ represent in terms of the words and/or sentences and the relationship between words.

If I think about matrices as linear transformations essentially SVD is taking some matrix $$X$$ and decomposing it into a (rotation)(stretch)(rotation).

I am ultimately looking for a way to think about the SVD decomposition of a co-occurrence matrix more intuitively.

• Welcome to the CV community. I tried to show you the interpretation of the document-by-term matrix because it is much more general from the co-occurrence matrix while encapsulating very similar information. Feb 1, 2019 at 22:44

1 Answer

I think that the co-occurrence matrix is not very relevant for an NLP application as the information it includes is readily available in the term-by-document count matrix.

Through the analysis of the document-by-term matrix what we would be describing would be a prototypical application of Latent Semantic Analysis (LSA) (some people refer to LSA as Latent Semantic Index (LSI) - it is the same thing). The original reference for LSA is Deerwester et al. (1990) "Indexing by latent semantic analysis" and it pretty much the application of PCA to a term by document count matrix. Usually one does not use the raw count but some transformed count through TFIDF (term frequency-inverse document frequency) but the idea remains the same.

In short, we can think of the $$U$$, $$\Sigma$$ and $$V$$ representing a "document-to-concept", a "power-of-concept" and a "term-to-concept" (assuming an $$m \times n$$ matrix $$A$$ with $$m$$ documents and $$n$$ terms) respectively.

Co-occurrence matrices can be very helpful for image analysis (e.g. see Clausi (2002) "An analysis of co-occurrence texture statistics as a function of grey level quantization" as they can be used to encapsulate texture statistics but that is because texture does not have the same interpretation as actual word-terms. (I recall seeing co-occurance data being used a metric for the purposes of MDS but again for an NLP we might as well use the term-by-document counts matrix to get the same info.)