The mutual information is usually calculated for random variables. I don't know what it exactly measures. Events are more intuitive to me. Can we use it for events, for example, cloudy sky and rainy weather? If yes, could you bring examples of events with high/low mutual information?

  • $\begingroup$ “what is the process of your thinking to find such examples” is highly subjective and doesn’t define an answerable question. $\endgroup$
    – Tim
    May 2 at 10:18
  • $\begingroup$ @Tim I simply meant give two typical examples, however, I now doubt if it applies to events or random variables and that is the first part of question. $\endgroup$
    – Ahmad
    May 2 at 16:31
  • $\begingroup$ have you tried Wikipedia? High mutual information: one variable is a deterministic function of the other, so if X is a dice throw, Y = X + 1. Low mutual information: two variables are independent, e.g. two independent coin tosses. This is what your question is about? $\endgroup$
    – Tim
    May 2 at 16:47
  • $\begingroup$ @Tim, I checkd but it's not intuative to me, I sent parts that I didn't understood. Also, being determinist function isn't a realistic and intuitive example. For example, consider these events: A: Cloudy sky B: Rainy weather or A: Summer B:rainy weather. Which pair has high and which has low mutual information... $\endgroup$
    – Ahmad
    May 2 at 18:20
  • 1
    $\begingroup$ You ask multiple different questions at once, please make it a single, focused question. $\endgroup$
    – Tim
    May 2 at 18:34

It sounds like you're looking for the pointwise mutual information (pmi), which relates two events. Mutual information averages the pmi over all possible events.

What this measures is whether two events tend to occur together more often you'd expect, just considering the events independently. If they occur more often than that, pmi is positive. Less often, it's negative. Conditionally independent, it's zero.

$$\operatorname{pmi}(x;y) \equiv \log\frac{p(x,y)}{p(x)p(y)} = \log\frac{p(x|y)}{p(x)} = \log\frac{p(y|x)}{p(y)}$$

I'll leave to your own intuitions the sort of events that do or don't have high pmi.

In the example you provide, $x$ might be the event of a cloudy sky, from a space of options $\mathcal{X} = \{\text{cloudy}, \text{not cloudy}\}$, and $y$ might be the event of rainy weather, from a space of options $\mathcal{Y} = \{\text{rainy}, \text{snowy}, \text{hazy}, \text{sunny}\}$. By averaging the pmi over all of these combinations, you get the mutual information.

  • 1
    $\begingroup$ pmi seems to be (a special case of) what statisticians call a log likelihood ratio. $\endgroup$ May 2 at 20:53
  • $\begingroup$ Yes, between the joint distribution and the individual distributions if they were independent. $\endgroup$ May 2 at 21:27
  • $\begingroup$ Thanks, it says " Note that even though PMI may be negative or positive, its expected outcome over all joint events (MI) is positive." Can you say why? $\endgroup$
    – Ahmad
    May 3 at 5:50
  • $\begingroup$ It’s provable with Jensen’s inequality. $\endgroup$ May 3 at 13:11

Mutual information is linked to the concept of entropy of a random variable that quantifies the expected "amount of information" held in a random variable, or the amount of uncertainty about the outcome of a random variable.

The key point is that it's about a random variable, such as $Weather$ or $Sky$. A random variable consists of a finite number of events or outcomes whose occurrence is by random with a probability. For example, outcomes of $Weather$ are:

$\mathcal{W} = \{\text{rainy}, \text{sunny}\}$

and outcomes of $Sky$ are:

$\mathcal{S} = \{\text{cloudy}, \text{half-cloudy}, \text{clear}\}$ each with a probability.

A random variable like $Weather$ has the highest entropy when all outcomes have equal probability. Then you can't predict that whether tomorrow is likely rainy or sunny, and any outcome is a surprise for you. For these two outcomes, we need $1$ bit (0 or 1) to report the weather of each day, which is the maximum entropy value in this case.

However, if $P(\text{sunny})$ is much higher than $P(\text{rainy})$, predicting tomorrow is $\text{sunny}$ has less information content compared to predicting that it's $\text{rainy}$. So, it's not a news for you if one says tomorrow is sunny.

The entropy for a variable averages the information content of all outcomes. If always is sunny, we have no entropy. We need zero bit to transfer information for every day.

The mutual information quantifies the amount of information or uncertainty that remains for a random variable like $Weather$ when we know the outcome of another variable like $Sky$. So if you know that $Sky=\text{clear}$ it affects the probabilities of $Weather$ outcomes. Then the probability of $\text{rainy}$ can decrease! It shows a type of dependence. It's a non-negative value and it's zero when two random variables are independent. It has various formulas, one is:

$$I(X;Y) = H(X) - H(X | Y)= H(Y) - H(Y|X)$$

$H(X)$ is the entropy of $X$ and $H(X | Y)$ is the conditional entropy of two variables $X$ and $Y$. When they are independent, then $H(X) = H(X|Y)$, which means knowing about $Y$ has no effect on the entropy of $X$.

Also as the accepted answer pointed out, there is a measure called pointwise mutual information, which measures the mutual information between two single events, such as rainy weather and cloudy sky. The mutual information is the expected value of PMI among pairs of outcomes of two random variables.

  • 1
    $\begingroup$ if you would like to answer your own question, please consider doing so while asking it, not after. You can do so by checking the "Answer your own question" box under the box where you enter your question. $\endgroup$
    – mhdadk
    May 3 at 10:33
  • 1
    $\begingroup$ @mhdadk thanks, no I reach conclusions after reading the other answers and related articles and then post my understandings for myself and others. $\endgroup$
    – Ahmad
    May 3 at 10:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.