10
$\begingroup$

I am a bit confused. Can someone explain to me how to calculate mutual information between two terms based on a term-document matrix with binary term occurrence as weights?

$$ \begin{matrix} & 'Why' & 'How' & 'When' & 'Where' \\ Document1 & 1 & 1 & 1 & 1 \\ Document2 & 1 & 0 & 1 & 0 \\ Document3 & 1 & 1 & 1 & 0 \end{matrix} $$

$$I(X;Y)= \sum_{y \in Y} \sum_{x \in X} p(x,y) \log\left(\frac{p(x,y)}{p(x)p(y)} \right)$$

Thank you

$\endgroup$
  • $\begingroup$ What are $X$ and $Y$? $\endgroup$ – Dilip Sarwate Dec 29 '12 at 15:41
  • $\begingroup$ X and Y are terms. X could be "Why" and Y could be "How". $\endgroup$ – user18075 Dec 29 '12 at 15:48
7
$\begingroup$

How about forming a joint probability table holding the normalized co-occurences in documents. Then you can obtain joint entropy and marginal entropies using the table. Finally, $$I(X,Y) = H(X)+H(Y)-H(X,Y). $$

$\endgroup$
  • 1
    $\begingroup$ When the joint and marginal distributions have been determined, why is it necessary to compute $H(X)$, $H(Y)$ and $H(X,Y)$ and use the formula you exhibit? Can't the mutual information be determined directly via the formula given by the OP since everything needed for "plugging in", viz. $p(x,y), p(x)$ and $p(y)$ are known at this point? $\endgroup$ – Dilip Sarwate Dec 29 '12 at 19:33
  • 2
    $\begingroup$ formulas are equivalent except the latter can be more interpretable at first glance. $\endgroup$ – Zoran Dec 29 '12 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.