I have been trying to implement mutual information in a java program that tries to identify terms that occur often together in documents. Basically, the program does the following: There is a text collection of just over 4.000.000 summaries from Wikipedia. This is searched through based on a query, resulting in a collection of say 100 texts that are deemed as a match to the query by Lucene index. I then create bigrams (structures of two words) so that all words are paired with all the other words in the result collection. Each bigram is then calculated a mutual information value for. The idea is that the highest scoring bigrams should be (hopefully) directly relevant to the original query.

On to how I understand the Mutual Information equation, which for the record looks like this:

P(x,y) * log(P(x,y) / (P(x) * P(y)),

where P(x,y) is the probability of term x and y occurring in a document, P(x) is the probability of just term x occurs in a document, and likewise for P(y) for term y.

I am unsure if I understand correctly what this means in terms of calculating the probability values. It is calculated by finding the frequency of a term divided by the number of articles in the result set. But is it the term frequency or the document frequency for a given term?

Ie. is P(x) in a result set of 100 articles the number of times term x occurs in the collection divided by 100? Or is it the number of articles in which term x occurs at least once divided by 100? And if it is the former, how then is P(x,y) calculated?


1 Answer 1


P(x) is the number of document where x occurs divided by the number of documents. If it was the number of occurences of x divided by 100, then P(x) could be higher than one, which is against the definition of a probability.

P(x,y) is the number of documents where both x and y occurs divided by the number of documents.

You can also have a look at this question on stackoverflow.

  • $\begingroup$ This is what I expected, but my councillor tells me that I need to include the term frequency of terms x and y as well, and I just cannot see how that is done based on the equation examples I see. They all basically support what you just said. When I asked him he just simply said that I should view the documents as sets or vectors, but that didn't really help me at all.. $\endgroup$
    – Geir K.H.
    Commented Feb 3, 2014 at 16:33
  • $\begingroup$ Your councillor answer is strange because considering document as a set of words would remove term frequency information. In a set, a word is present only once. $\endgroup$ Commented Feb 4, 2014 at 13:45
  • $\begingroup$ I believe you need to use a kernel density estimator and calculate P(x) from there. meaning first you estimate the probability density function of your vector x and then calculating P(X) from this PDF. This P(X), is not the frequency of your word in the document, but is the probability of occurring P(X) in all documents. $\endgroup$
    – Areza
    Commented May 7, 2014 at 5:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.