# Computing mutual information

I have a problem when computing the mutual information between two variables. Let's consider the following table:

             k=1       k=2       k=3
c = 1         10        20         5
c = 2          5         3        20


I want to calculate the mutual information between the class ($c$) and the cluster ($k$).

E.g., for $\text{MI}(c=1, k=1)$, I will calculate the $P(c,k)$ as follows:

             k=1            k~=1
c=1        10/63           25/63
c~=1        5/63           23/63


and I can calculate the MI based on the following:

$\text{MI}(c,k) = \sum_{c\in\{0,1\}} \sum_{k\in\{0,1\}} p(c,k)log_2 \frac{p(c,k)}{p(c)p(k)}$

In this case: I am getting the same MI for $(c=1, k=1)$ and $(c=2, k=1)$. What is the reason for this? How can I calculate the MI for each $(c,k)$ based on the first table?

MI is between two random variables. In this case, $c$ and $k$ are random variables.

If you take $c=1$, it isn't really random anymore, and MI should be zero. Likewise for $k=1$.

(Also, one more thing to keep in mind is that your 'plug-in' estimation of MI is biased.)

EDIT: MI between $I(c=1)$ vs $I(k=1)$ where $I$ is the indicator function, is a different matter. Your double usage of the same variable confused me.

Now your $c$ only take 2 values, therefore, $I(c=1) = 1 - I(c=2)$. MI is invariant invertible transformation of variables, that's why $MI(I(c=1),I(k=1)) = MI(I(c=2),I(k=1))$.

• thanks for your reply. But how can I find the dependency between (c=1, k=1) and (c=2, k=1) ? – user570593 Oct 12 '13 at 14:12
• what do you mean by (c=1,k=1)? is that a random quantity? – Memming Oct 12 '13 at 14:15
• For example I want to find the dependency between the cluster 1 (k=1) and a class (c=1). – user570593 Oct 12 '13 at 14:21
• what is random in $k=1$? It's always $k=1$. You can't quantify a statistical dependence if there's no randomness. – Memming Oct 12 '13 at 14:40
• K and c are random variables here. I want to calculate the dependecy between a particular k and particular c. how to do it with MI? – user570593 Oct 12 '13 at 14:41