I have a problem when computing the mutual information between two variables. Let's consider the following table:

             k=1       k=2       k=3    
c = 1         10        20         5    
c = 2          5         3        20

I want to calculate the mutual information between the class ($c$) and the cluster ($k$).

E.g., for $\text{MI}(c=1, k=1)$, I will calculate the $P(c,k)$ as follows:

             k=1            k~=1    
c=1        10/63           25/63    
c~=1        5/63           23/63

and I can calculate the MI based on the following:

$\text{MI}(c,k) = \sum_{c\in\{0,1\}} \sum_{k\in\{0,1\}} p(c,k)log_2 \frac{p(c,k)}{p(c)p(k)}$

In this case: I am getting the same MI for $(c=1, k=1)$ and $(c=2, k=1)$. What is the reason for this? How can I calculate the MI for each $(c,k)$ based on the first table?


1 Answer 1


MI is between two random variables. In this case, $c$ and $k$ are random variables.

If you take $c=1$, it isn't really random anymore, and MI should be zero. Likewise for $k=1$.

(Also, one more thing to keep in mind is that your 'plug-in' estimation of MI is biased.)

EDIT: MI between $I(c=1)$ vs $I(k=1)$ where $I$ is the indicator function, is a different matter. Your double usage of the same variable confused me.

Now your $c$ only take 2 values, therefore, $I(c=1) = 1 - I(c=2)$. MI is invariant invertible transformation of variables, that's why $MI(I(c=1),I(k=1)) = MI(I(c=2),I(k=1))$.

  • $\begingroup$ thanks for your reply. But how can I find the dependency between (c=1, k=1) and (c=2, k=1) ? $\endgroup$
    – user570593
    Commented Oct 12, 2013 at 14:12
  • $\begingroup$ what do you mean by (c=1,k=1)? is that a random quantity? $\endgroup$
    – Memming
    Commented Oct 12, 2013 at 14:15
  • $\begingroup$ For example I want to find the dependency between the cluster 1 (k=1) and a class (c=1). $\endgroup$
    – user570593
    Commented Oct 12, 2013 at 14:21
  • $\begingroup$ what is random in $k=1$? It's always $k=1$. You can't quantify a statistical dependence if there's no randomness. $\endgroup$
    – Memming
    Commented Oct 12, 2013 at 14:40
  • $\begingroup$ K and c are random variables here. I want to calculate the dependecy between a particular k and particular c. how to do it with MI? $\endgroup$
    – user570593
    Commented Oct 12, 2013 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.