4
$\begingroup$

This question already has an answer here:

Why should I use KL divergence over just giving the abs difference from two PDFs?

$\endgroup$

marked as duplicate by kjetil b halvorsen, Frans Rodenburg, whuber Apr 30 at 17:54

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1
$\begingroup$

Kullback-Leibler (and other information theoretic divergences, the $f$-divergences) is linked to the Fisher Information: KL is locally an approximation of the Fisher-Rao geodesic distance, which is a Riemannian metric parameterized by the Fisher Information matrix. It has the property to be the only Riemannian metric (up to a multiplicative factor) which is invariant to reparameterization of the parameter space (in case of a parametric distribution).

Also, and this is important here, the divergence KL (and $f$-divergences) can be seen as diverging by the 'right' amount with respect to 'information' / statistical uncertainty: if you have a parametric distribution, paramaterized by $\theta$, and you estimate $\hat{\theta}$ with an unbiased estimator, then the Cramer-Rao lower bound tells you that $\mathrm{var}(\hat{\theta}) \geq \frac{1}{I(\theta)}$, where $I$ is the Fisher Information matrix parameterizing the KL divergence $\mathrm{KL}(\theta \| \theta + d\theta) \approx \frac{1}{2}\theta^\top I(\theta) \theta$.

Information Geometry is the field which investigates this kind of questions.

If you require a symmetry, you either symmetrized KL, it is called the Jeffreys divergence, or use Hellinger (which is also a $f$-divergence and a proper metric distance).

$\endgroup$
1
$\begingroup$

The empirical value of KL divergence is what maximum likelihood estimation tries to minimize. That is, $\max_\theta \sum_{i=1}^n \log p(X_i|\theta)$ is equivalent to $\min_\theta \sum_{i=1}^n \log \frac{q(X_i)}{p(X_i|\theta)}$ (under regularity conditions).

If you wonder why this is related to KL divergence, note that $KL(q || p_\theta) := E_q[\log \frac{q(X)}{p(X|\theta)}]$. Here it is assumed that $X_i$'s are i.i.d. samples from the distribution $q$.

And maximum likelihood estimators have good properties.

(In response to another answer)

@mic (I can't comment yet, so let me add this here)

It has the property to be the only Riemannian metric (up to a multiplicative factor) which is invariant to reparameterization of the parameter space (in case of a parametric distribution).

Could you add a reference to support this? Because as far as I understand, this property has been proved only for the space of finite-dimensional, discrete distributions. (Well, maybe this is due to my limited knowledge, so one should check out anyway)

Edit

From an engineering point of view, there are alternatives to KL divergence too. One such an example is called Pearson divergence.

$\endgroup$