Given sentences

  1. I enjoy flying.
  2. I like NLP.
  3. I like deep learning

We can form a Co-Occurrence Matrix as follows:enter image description here

Now we can apply Singular Value Decomposition to this matrix to get $X = U \Sigma V^T$ where $U$ and $V$ are orthogonal and $\Sigma$ is diagonal and the singular values are sorted (I believe) so $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_r$.

My question is what do the matrices $U$, $\Sigma$, $V$ represent in terms of the words and/or sentences and the relationship between words.

If I think about matrices as linear transformations essentially SVD is taking some matrix $X$ and decomposing it into a (rotation)(stretch)(rotation).

I am ultimately looking for a way to think about the SVD decomposition of a co-occurrence matrix more intuitively.

  • $\begingroup$ Welcome to the CV community. I tried to show you the interpretation of the document-by-term matrix because it is much more general from the co-occurrence matrix while encapsulating very similar information. $\endgroup$
    – usεr11852
    Feb 1, 2019 at 22:44

1 Answer 1


I think that the co-occurrence matrix is not very relevant for an NLP application as the information it includes is readily available in the term-by-document count matrix.

Through the analysis of the document-by-term matrix what we would be describing would be a prototypical application of Latent Semantic Analysis (LSA) (some people refer to LSA as Latent Semantic Index (LSI) - it is the same thing). The original reference for LSA is Deerwester et al. (1990) "Indexing by latent semantic analysis" and it pretty much the application of PCA to a term by document count matrix. Usually one does not use the raw count but some transformed count through TFIDF (term frequency-inverse document frequency) but the idea remains the same.

In short, we can think of the $U$, $\Sigma$ and $V$ representing a "document-to-concept", a "power-of-concept" and a "term-to-concept" (assuming an $m \times n$ matrix $A$ with $m$ documents and $n$ terms) respectively.

Co-occurrence matrices can be very helpful for image analysis (e.g. see Clausi (2002) "An analysis of co-occurrence texture statistics as a function of grey level quantization" as they can be used to encapsulate texture statistics but that is because texture does not have the same interpretation as actual word-terms. (I recall seeing co-occurance data being used a metric for the purposes of MDS but again for an NLP we might as well use the term-by-document counts matrix to get the same info.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.