I have a problem when computing the mutual information between two variables. Let's consider the following table:

             k=1       k=2       k=3    
c = 1         10        20         5    
c = 2          5         3        20

I want to calculate the mutual information between the class ($c$) and the cluster ($k$).

E.g., for $\text{MI}(c=1, k=1)$, I will calculate the $P(c,k)$ as follows:

             k=1            k~=1    
c=1        10/63           25/63    
c~=1        5/63           23/63

and I can calculate the MI based on the following:

$\text{MI}(c,k) = \sum_{c\in\{0,1\}} \sum_{k\in\{0,1\}} p(c,k)log_2 \frac{p(c,k)}{p(c)p(k)}$

In this case: I am getting the same MI for $(c=1, k=1)$ and $(c=2, k=1)$. What is the reason for this? How can I calculate the MI for each $(c,k)$ based on the first table?


MI is between two random variables. In this case, $c$ and $k$ are random variables.

If you take $c=1$, it isn't really random anymore, and MI should be zero. Likewise for $k=1$.

(Also, one more thing to keep in mind is that your 'plug-in' estimation of MI is biased.)

EDIT: MI between $I(c=1)$ vs $I(k=1)$ where $I$ is the indicator function, is a different matter. Your double usage of the same variable confused me.

Now your $c$ only take 2 values, therefore, $I(c=1) = 1 - I(c=2)$. MI is invariant invertible transformation of variables, that's why $MI(I(c=1),I(k=1)) = MI(I(c=2),I(k=1))$.

  • $\begingroup$ thanks for your reply. But how can I find the dependency between (c=1, k=1) and (c=2, k=1) ? $\endgroup$ – user570593 Oct 12 '13 at 14:12
  • $\begingroup$ what do you mean by (c=1,k=1)? is that a random quantity? $\endgroup$ – Memming Oct 12 '13 at 14:15
  • $\begingroup$ For example I want to find the dependency between the cluster 1 (k=1) and a class (c=1). $\endgroup$ – user570593 Oct 12 '13 at 14:21
  • $\begingroup$ what is random in $k=1$? It's always $k=1$. You can't quantify a statistical dependence if there's no randomness. $\endgroup$ – Memming Oct 12 '13 at 14:40
  • $\begingroup$ K and c are random variables here. I want to calculate the dependecy between a particular k and particular c. how to do it with MI? $\endgroup$ – user570593 Oct 12 '13 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.