# KL divergence vs Absolute Difference between two distributions? [duplicate]

Why should I use KL divergence over just giving the abs difference from two PDFs?

• BTW, can you tell more about what is "absolute difference" for two PDFs? – Haitao Du Jul 26 '16 at 17:00
• – kjetil b halvorsen Aug 9 '17 at 14:59

Kullback-Leibler (and other information theoretic divergences, the $f$-divergences) is linked to the Fisher Information: KL is locally an approximation of the Fisher-Rao geodesic distance, which is a Riemannian metric parameterized by the Fisher Information matrix. It has the property to be the only Riemannian metric (up to a multiplicative factor) which is invariant to reparameterization of the parameter space (in case of a parametric distribution).

Also, and this is important here, the divergence KL (and $f$-divergences) can be seen as diverging by the 'right' amount with respect to 'information' / statistical uncertainty: if you have a parametric distribution, paramaterized by $\theta$, and you estimate $\hat{\theta}$ with an unbiased estimator, then the Cramer-Rao lower bound tells you that $\mathrm{var}(\hat{\theta}) \geq \frac{1}{I(\theta)}$, where $I$ is the Fisher Information matrix parameterizing the KL divergence $\mathrm{KL}(\theta \| \theta + d\theta) \approx \frac{1}{2}\theta^\top I(\theta) \theta$.

Information Geometry is the field which investigates this kind of questions.

If you require a symmetry, you either symmetrized KL, it is called the Jeffreys divergence, or use Hellinger (which is also a $f$-divergence and a proper metric distance).

The empirical value of KL divergence is what maximum likelihood estimation tries to minimize. That is, $\max_\theta \sum_{i=1}^n \log p(X_i|\theta)$ is equivalent to $\min_\theta \sum_{i=1}^n \log \frac{q(X_i)}{p(X_i|\theta)}$ (under regularity conditions).

If you wonder why this is related to KL divergence, note that $KL(q || p_\theta) := E_q[\log \frac{q(X)}{p(X|\theta)}]$. Here it is assumed that $X_i$'s are i.i.d. samples from the distribution $q$.

And maximum likelihood estimators have good properties.

## (In response to another answer)

@mic (I can't comment yet, so let me add this here)

It has the property to be the only Riemannian metric (up to a multiplicative factor) which is invariant to reparameterization of the parameter space (in case of a parametric distribution).

Could you add a reference to support this? Because as far as I understand, this property has been proved only for the space of finite-dimensional, discrete distributions. (Well, maybe this is due to my limited knowledge, so one should check out anyway)

## Edit

From an engineering point of view, there are alternatives to KL divergence too. One such an example is called Pearson divergence.