# How to calculate mutual information?

I am a bit confused. Can someone explain to me how to calculate mutual information between two terms based on a term-document matrix with binary term occurrence as weights?

$$\begin{matrix} & 'Why' & 'How' & 'When' & 'Where' \\ Document1 & 1 & 1 & 1 & 1 \\ Document2 & 1 & 0 & 1 & 0 \\ Document3 & 1 & 1 & 1 & 0 \end{matrix}$$

$$I(X;Y)= \sum_{y \in Y} \sum_{x \in X} p(x,y) \log\left(\frac{p(x,y)}{p(x)p(y)} \right)$$

Thank you

• What are $X$ and $Y$? – Dilip Sarwate Dec 29 '12 at 15:41
• X and Y are terms. X could be "Why" and Y could be "How". – user18075 Dec 29 '12 at 15:48

How about forming a joint probability table holding the normalized co-occurences in documents. Then you can obtain joint entropy and marginal entropies using the table. Finally, $$I(X,Y) = H(X)+H(Y)-H(X,Y).$$
• When the joint and marginal distributions have been determined, why is it necessary to compute $H(X)$, $H(Y)$ and $H(X,Y)$ and use the formula you exhibit? Can't the mutual information be determined directly via the formula given by the OP since everything needed for "plugging in", viz. $p(x,y), p(x)$ and $p(y)$ are known at this point? – Dilip Sarwate Dec 29 '12 at 19:33