# Natural Language Processing: Basic Dimension Reduction with SVD of a Co-Occurence Matrix

Given sentences

1. I enjoy flying.
2. I like NLP.
3. I like deep learning

We can form a Co-Occurrence Matrix as follows:

Now we can apply Singular Value Decomposition to this matrix to get $$X = U \Sigma V^T$$ where $$U$$ and $$V$$ are orthogonal and $$\Sigma$$ is diagonal and the singular values are sorted (I believe) so $$\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_r$$.

My question is what do the matrices $$U$$, $$\Sigma$$, $$V$$ represent in terms of the words and/or sentences and the relationship between words.

If I think about matrices as linear transformations essentially SVD is taking some matrix $$X$$ and decomposing it into a (rotation)(stretch)(rotation).

I am ultimately looking for a way to think about the SVD decomposition of a co-occurrence matrix more intuitively.

• Welcome to the CV community. I tried to show you the interpretation of the document-by-term matrix because it is much more general from the co-occurrence matrix while encapsulating very similar information. – usεr11852 Feb 1 '19 at 22:44

In short, we can think of the $$U$$, $$\Sigma$$ and $$V$$ representing a "document-to-concept", a "power-of-concept" and a "term-to-concept" (assuming an $$m \times n$$ matrix $$A$$ with $$m$$ documents and $$n$$ terms) respectively.