KL divergence vs Absolute Difference between two distributions? [duplicate]

Why should I use KL divergence over just giving the abs difference from two PDFs?

marked as duplicate by kjetil b halvorsen, Frans Rodenburg, whuber♦Apr 30 at 17:54

• BTW, can you tell more about what is "absolute difference" for two PDFs? – hxd1011 Jul 26 '16 at 17:00
• – kjetil b halvorsen Aug 9 '17 at 14:59

Kullback-Leibler (and other information theoretic divergences, the $f$-divergences) is linked to the Fisher Information: KL is locally an approximation of the Fisher-Rao geodesic distance, which is a Riemannian metric parameterized by the Fisher Information matrix. It has the property to be the only Riemannian metric (up to a multiplicative factor) which is invariant to reparameterization of the parameter space (in case of a parametric distribution).

Also, and this is important here, the divergence KL (and $f$-divergences) can be seen as diverging by the 'right' amount with respect to 'information' / statistical uncertainty: if you have a parametric distribution, paramaterized by $\theta$, and you estimate $\hat{\theta}$ with an unbiased estimator, then the Cramer-Rao lower bound tells you that $\mathrm{var}(\hat{\theta}) \geq \frac{1}{I(\theta)}$, where $I$ is the Fisher Information matrix parameterizing the KL divergence $\mathrm{KL}(\theta \| \theta + d\theta) \approx \frac{1}{2}\theta^\top I(\theta) \theta$.

Information Geometry is the field which investigates this kind of questions.

If you require a symmetry, you either symmetrized KL, it is called the Jeffreys divergence, or use Hellinger (which is also a $f$-divergence and a proper metric distance).

The empirical value of KL divergence is what maximum likelihood estimation tries to minimize. That is, $\max_\theta \sum_{i=1}^n \log p(X_i|\theta)$ is equivalent to $\min_\theta \sum_{i=1}^n \log \frac{q(X_i)}{p(X_i|\theta)}$ (under regularity conditions).

If you wonder why this is related to KL divergence, note that $KL(q || p_\theta) := E_q[\log \frac{q(X)}{p(X|\theta)}]$. Here it is assumed that $X_i$'s are i.i.d. samples from the distribution $q$.

And maximum likelihood estimators have good properties.