# How to calculate mutual information from frequencies

Can someone explain to me how to calculate mutual information from contingency table?

I have a contingency table containing counts from a sample of data

And I want to calculate the mutual information between motif and condition. Since the mutual information formula requires probabilities, how can I estimate it from frequencies? Or how to obtain the mutual information distribution?

I'm not a stats specialist, but I will give it a shot.

First, we can approximate the probability of each event by its empirical probability, i.e. the number of occurrences divided by the total number of trials:

$p(motif_i, condition_j) = \frac{\text{number of occurrences of motif i with condition j}}{ \sum_{i,j} \text{number of occurrences of motif i with condition j}}$

I'll use the shorthands m_1, m_2, c_1, c_2 for motifs and conditions in your table. The approximation gives the following joint distribution $p(m_i,c_j)$:

     c_1  c_2
m_1  0.1 0.05
m_2  0.4 0.45


Marginal probabilities can be computed by just summing rows and columns. Have a look at the example there: https://en.wikipedia.org/wiki/Marginal_distribution I.e. here, $p(m_1)=0.15$ and $p(c_1)=0.5$.

Then, the mutual information can be computed from its definition:

$I(motif;condition) = \sum_{i \in [1,2], j \in [1,2]} p(m_i,c_j)\log(\frac{p(m_i,c_j)}{p(m_i)p(c_j)})$

• Pay attention. This is not the mutual information, but an estimation based on sampled data, which is positively biased compared to the mutual information of the distribution from which the samples are drawn. Sep 4, 2019 at 12:23