Why does mutual information use KL divergence? Mutual information between a pair of random variables $X,Y$ having joint distribution $P_{(X,Y)}$ and marginal distributions $P_X,P_Y$ respectively is defined as
$$I(X,Y)\equiv D_{\text{KL}}(P_{(X,Y)}\|P_X\otimes P_Y ),$$
where $D_{\text{KL}}$ is the KL divergence. Intuitively, this measures how much "information" is revealed about one random variable through observing the other by quantifying how far the joint distribution is from the product of marginals (this distance being zero when $X,Y$ are independent).
Why not more flexibly allow for other notions of statistical distance? i.e. Why not define
$$\tilde I(X,Y,d)\equiv d(P_{(X,Y)},P_X\otimes P_Y )$$
for arbitrary distance $d$? There are distances that are at least as compelling as KL divergence, such as Jensen-Shannon divergence, which at least symmetrizes KL divergence, or
the Wasserstein metric, which is actually a metric and enjoys other attractive properties (as observed in the ML literature).
I understand mutual information as defined has connections with entropy, so perhaps this makes the definition tractable? What merits are there in using the KL divergence vs. other distances?
 A: I think that one advantage of $KL$ distance is that is has a simple interpretation, according to wiki $D_{KL}(P, Q)$ is "the expected excess surprise from using $Q$ as a model when the actual distribution is $P$". Also it is easy to compute. On the contrast, the Wasserstein metric is defined as the solution to some linear optimization problem. One can solve it effectively but it definitely longer compared to using a simple formula. The symmetrized $KL$ divergence is super similar but its interpretation is just a bit less straightforward: "The Jensen–Shannon divergence is the mutual information between a random variable $X$ associated to a mixture distribution between $P$ and $Q$ and the binary indicator variable $Z$ that is used to switch between $P$ and $Q$ to produce the mixture." I didn't look up all the items on the list of statistical distances but some of them are definitely harder to compute compared to $KL$ distance. Also $KL$ distance is easy to differentiate, and that is great for any applications in machine learning (using as a loss function).
