Natural Language Processing: Basic Dimension Reduction with SVD of a Co-Occurence Matrix Given sentences 


*

*I enjoy flying.

*I like NLP.

*I like deep learning


We can form a Co-Occurrence Matrix as follows:
Now we can apply Singular Value Decomposition to this matrix to get
$X = U \Sigma V^T$ where $U$ and $V$ are orthogonal and $\Sigma$ is diagonal and the singular values are sorted (I believe) so $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_r$.
My question is what do the matrices $U$, $\Sigma$, $V$ represent in terms of the words and/or sentences and the relationship between words. 
If I think about matrices as linear transformations essentially SVD is taking some matrix $X$ and decomposing it into a (rotation)(stretch)(rotation). 
I am ultimately looking for a way to think about the SVD decomposition of a co-occurrence matrix more intuitively. 
 A: I think that the co-occurrence matrix is not very relevant for an NLP application as the information it includes is readily available in the term-by-document count matrix. 
Through the analysis of the document-by-term matrix what we would be describing would be a prototypical application of Latent Semantic Analysis (LSA) (some people refer to LSA as Latent Semantic Index (LSI) - it is the same thing). The original reference for LSA is Deerwester et al. (1990) "Indexing by latent semantic analysis" and it pretty much the application of  PCA to a term by document count matrix. Usually one does not use the raw count but some transformed count through TFIDF (term frequency-inverse document frequency) but the idea remains the same. 
In short, we can think of the $U$, $\Sigma$ and $V$ representing a "document-to-concept", a "power-of-concept" and a "term-to-concept" (assuming an $m \times n$ matrix $A$ with $m$ documents and $n$ terms) respectively.
Co-occurrence matrices can be very helpful for image analysis (e.g. see Clausi (2002) "An analysis of co-occurrence texture statistics as a function of grey level quantization" as they can be used to encapsulate texture statistics but that is because texture does not have the same interpretation as actual word-terms. (I  recall seeing co-occurance data being used a metric for the purposes of MDS but again for an NLP we might as well use the term-by-document counts matrix to get the same info.)
