3
$\begingroup$

Mutual information between a pair of random variables $X,Y$ having joint distribution $P_{(X,Y)}$ and marginal distributions $P_X,P_Y$ respectively is defined as

$$I(X,Y)\equiv D_{\text{KL}}(P_{(X,Y)}\|P_X\otimes P_Y ),$$

where $D_{\text{KL}}$ is the KL divergence. Intuitively, this measures how much "information" is revealed about one random variable through observing the other by quantifying how far the joint distribution is from the product of marginals (this distance being zero when $X,Y$ are independent).

Why not more flexibly allow for other notions of statistical distance? i.e. Why not define

$$\tilde I(X,Y,d)\equiv d(P_{(X,Y)},P_X\otimes P_Y )$$

for arbitrary distance $d$? There are distances that are at least as compelling as KL divergence, such as Jensen-Shannon divergence, which at least symmetrizes KL divergence, or the Wasserstein metric, which is actually a metric and enjoys other attractive properties (as observed in the ML literature).

I understand mutual information as defined has connections with entropy, so perhaps this makes the definition tractable? What merits are there in using the KL divergence vs. other distances?

$\endgroup$
4
  • 2
    $\begingroup$ This question appears circular and tautological: MI is whatever it is defined to be. If you want to go further, you need to articulate some kind of definition or criterion of what you understand "information" is. $\endgroup$
    – whuber
    Feb 9, 2022 at 18:08
  • $\begingroup$ @whuber Thanks for the comment. My question is more about the motivation of using KL divergence. Definitions are definitions, but I am interested to what extent other distances have been used in the literature to quantify information, for instance. If not, why not? We use different methods to quantify spread, different kernels in kernel density estimation, etc., so why not use different methods to quantify information, particularly when there is an immediate generalization as above? $\endgroup$ Feb 9, 2022 at 18:28
  • 1
    $\begingroup$ That might be answerable if we could have some characterization of what you mean by "information." Otherwise, who will know what we are talking about? Every person can approach such a question with their own personal understanding of "information." $\endgroup$
    – whuber
    Feb 9, 2022 at 20:10
  • $\begingroup$ The above expression is only one way to define - or more correctly, express - mutual information. The use of KLD to express MI seems to be increasing in popularity due to, dare I say, the hegemony of KLD within information theory. KLD is an entropic relationship (a divergence) between two variables. It does not, by itself, define mutual information. The ability to relate KLD to MI is due to the algebraic relationships of various entropic identities between two variables. See Equation 1.4 in "Cover and Thomas (1991) Elements of Information Theory" for the classical definition of MI. $\endgroup$
    – Mari153
    Jan 6 at 6:53

1 Answer 1

1
$\begingroup$

I think that one advantage of $KL$ distance is that is has a simple interpretation, according to wiki $D_{KL}(P, Q)$ is "the expected excess surprise from using $Q$ as a model when the actual distribution is $P$". Also it is easy to compute. On the contrast, the Wasserstein metric is defined as the solution to some linear optimization problem. One can solve it effectively but it definitely longer compared to using a simple formula. The symmetrized $KL$ divergence is super similar but its interpretation is just a bit less straightforward: "The Jensen–Shannon divergence is the mutual information between a random variable $X$ associated to a mixture distribution between $P$ and $Q$ and the binary indicator variable $Z$ that is used to switch between $P$ and $Q$ to produce the mixture." I didn't look up all the items on the list of statistical distances but some of them are definitely harder to compute compared to $KL$ distance. Also $KL$ distance is easy to differentiate, and that is great for any applications in machine learning (using as a loss function).

$\endgroup$
8
  • $\begingroup$ I always understood this phrase "expected excess surprise" to be a characterization of the otherwise undefined, qualitative term "surprise" rather than being any kind of motivation or explanation of the KL distance. Are you suggesting "surprise" has some kind of clear, quantitative characterization separate from this context? What is it? $\endgroup$
    – whuber
    Feb 9, 2022 at 21:44
  • $\begingroup$ Let's consider the simpler notion of an entropy. Entropy is "the minimum number of bits it would take us to encode our information". That seems concrete enough for me. $KL$ distance is an analogue of this. It is "how many bits of information we expect to lose". You can find more details here countbayesie.com/blog/2017/5/9/…. Hope that makes sense. $\endgroup$ Feb 9, 2022 at 21:52
  • $\begingroup$ I know perfectly well what entropy is and how people think of it--I was hoping to be enlightened about "surprise." But don't you think that the information in your comment would be a better explanation than the one in your answer? $\endgroup$
    – whuber
    Feb 9, 2022 at 21:57
  • $\begingroup$ I agree, not sure if such editing is allowed here. $\endgroup$ Feb 9, 2022 at 22:00
  • 1
    $\begingroup$ Such editing is not only allowed, it is encouraged: edit away! $\endgroup$
    – whuber
    Feb 9, 2022 at 22:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.