10
$\begingroup$

I am a bit confused. Can someone explain to me how to calculate mutual information between two terms based on a term-document matrix with binary term occurrence as weights?

$$ \begin{matrix} & 'Why' & 'How' & 'When' & 'Where' \\ Document1 & 1 & 1 & 1 & 1 \\ Document2 & 1 & 0 & 1 & 0 \\ Document3 & 1 & 1 & 1 & 0 \end{matrix} $$

$$I(X;Y)= \sum_{y \in Y} \sum_{x \in X} p(x,y) \log\left(\frac{p(x,y)}{p(x)p(y)} \right)$$

Thank you

$\endgroup$
2
  • $\begingroup$ What are $X$ and $Y$? $\endgroup$ Commented Dec 29, 2012 at 15:41
  • $\begingroup$ X and Y are terms. X could be "Why" and Y could be "How". $\endgroup$
    – user18075
    Commented Dec 29, 2012 at 15:48

1 Answer 1

7
$\begingroup$

How about forming a joint probability table holding the normalized co-occurences in documents. Then you can obtain joint entropy and marginal entropies using the table. Finally, $$I(X,Y) = H(X)+H(Y)-H(X,Y). $$

$\endgroup$
2
  • 1
    $\begingroup$ When the joint and marginal distributions have been determined, why is it necessary to compute $H(X)$, $H(Y)$ and $H(X,Y)$ and use the formula you exhibit? Can't the mutual information be determined directly via the formula given by the OP since everything needed for "plugging in", viz. $p(x,y), p(x)$ and $p(y)$ are known at this point? $\endgroup$ Commented Dec 29, 2012 at 19:33
  • 2
    $\begingroup$ formulas are equivalent except the latter can be more interpretable at first glance. $\endgroup$
    – Zoran
    Commented Dec 29, 2012 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.