Unless I misunderstand something, the following points are true:
The entropy of a variable is the average information that you get from it with each trial.
The mutual information between two variables is the average amount by which one variable's entropy is reduced when we learn the value/outcome of the other variable.
The pointwise mutual information between two specific outcomes of 2 variables is the amount by which one outcome's information is reduced (if we were to observe it) when we learn observe the other outcome (i.e. it is the information shared by those two outcomes).
So, since it seems that the term "entropy" is used to refer to expected information, then shouldn't "mutual information" be called "mutual entropy"?
This image would seem to make more sense if it were called mutual entropy, since it's the "overlap" between the two entropies:
Another way to summarise my current (potentially very flawed) understanding:
- Distributions have entropy
- Outcomes have information
And it seems like mutual information is a property of two distributions, not of two outcomes.
The reason I'm asking this is not because I'd be surprised if it were a somewhat misleading name (due to historical reasons, or whatever), but because it seems like a potential piece of evidence that I don't properly understand what mutual information, entropy and pointwise mutual information actually are. Thanks!