When looking at the eigenvectors of the covariance matrix, we get the directions of maximum variance (the first eigenvector is the direction in which the data varies the most, etc.); this is called principal component analysis (PCA).

I was wondering what it would mean to look at the eigenvectors/values of the mutual information matrix, would they point in the direction of maximum entropy?

  • 5
    $\begingroup$ I don't know, but have just learned that mutual information matrices are not always positive semi-definite: arxiv.org/abs/1307.6673. $\endgroup$
    – amoeba
    Nov 5 '15 at 21:44
  • 3
    $\begingroup$ It reminds something we worked on: people.eng.unimelb.edu.au/baileyj/papers/frp0038-Vinh.pdf $\endgroup$
    – Simone
    Nov 11 '15 at 6:19
  • $\begingroup$ the answer below doesn't address the question since it doesn't even mention eigenvectors. if the matrix is not positive semi-definite, then what? what does that have to do with eigenvectors $\endgroup$
    – develarist
    Dec 20 '20 at 8:05

While it is not a direct answer (as it is about pointwise mutual information), look at paper relating word2vec to a singular value decomposition of PMI matrix:

We analyze skip-gram with negative-sampling (SGNS), a word embedding method introduced by Mikolov et al., and show that it is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) of the respective word and context pairs, shifted by a global constant. We find that another embedding method, NCE, is implicitly factorizing a similar matrix, where each cell is the (shifted) log conditional probability of a word given its context. We show that using a sparse Shifted Positive PMI word-context matrix to represent words improves results on two word similarity tasks and one of two analogy tasks. When dense low-dimensional vectors are preferred, exact factorization with SVD can achieve solutions that are at least as good as SGNS’s solutions for word similarity tasks. On analogy questions SGNS remains superior to SVD. We conjecture that this stems from the weighted nature of SGNS’s factorization.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.