2
$\begingroup$

The KL divergence is not a metric because e.g. it does not satisfy the symmetry property that metrics posses. According to the definition of measure, the KL divergence doesn't seem to be a measure, although the related Wikipedia article introduces the KL divergence as "a measure of how one probability distribution is different from a second". So, mathematically, what is the KL divergence?

Note that I know how the KL divergence is defined, so don't tell me how it's defined.

$\endgroup$
3
  • 2
    $\begingroup$ It is a measure in the English sense. You can use the word the way the Wikipedia article does. You just have to avoid confusing measure theorists. :) $\endgroup$
    – hobbs
    May 13, 2020 at 0:49
  • 2
    $\begingroup$ It is a (directed) divergence, see en.wikipedia.org/wiki/Divergence_(statistics) $\endgroup$ Dec 5, 2023 at 2:12
  • 1
    $\begingroup$ (Divergence).. as mentioned in 'the related Wikipedia article'. $\endgroup$
    – seanv507
    Dec 5, 2023 at 8:35

2 Answers 2

5
$\begingroup$

The KL divergence is a pre-metric, given that it satisfies the two properties of pre-metrics

  1. $d(x, y) \geq 0, \forall x, y$
  2. $d(x, x) = 0, \forall x$

So, given that any pre-metric induces a topology, so does the KL divergence.

The KL divergence is related to other metrics (such as the Fisher information metric) or information-theoretic concepts (such as information content, mutual information, Shannon entropy, conditional entropy, and cross-entropy). See also the related Wikipedia page.

$\endgroup$
0
1
$\begingroup$

It is a divergence, a function that maps two probability densities (or mass function), $p, q$ into $[0, \infty]$ with many special properties.

In particular, KL is a Bregman divergence! Let $\Omega$ be the set of probability densities over your sample space $X$. Choose any function $F:\Omega\to \mathbb{R}$ that is continuously differentiable and strictly convex, and the function $D_F: \Omega^2 \to \mathbb{R}, \ \ D(p, q) = F(p)-F(q)-\langle \nabla F(q), p-q\rangle $ is a Bregman divergence.

Bregman divergences have many great properties such as being non-negative, $D_F(p, q) = 0$ iff $p=q$, convex in the first argument, ... (see Wikipedia page).

In particular if you set $F(p) = -\log(p)$, the induced Bregman divergence is indeed KL divergence! Hence, all the properties on that Wikipedia link apply.

Lastly note that as you mentioned, KL divergence is not a metric b/c it doesn't satisfy the triangle inequality. It is not a measure b/c it does not act on sets. It is a divergence.

$\endgroup$
4
  • $\begingroup$ ... and has the property that if the distributions are identical, it maps them into $0$. $\endgroup$
    – jbowman
    Dec 5, 2023 at 2:06
  • 1
    $\begingroup$ While not wrong, this is not particularly helpful $\endgroup$ Dec 5, 2023 at 2:07
  • $\begingroup$ Agreed. The basic issue is that plenty of such functions are not divergences. It's kind of like trying to define "mammal" as a "terrestrial life form:" you get a tiny bit of information but not much enlightenment (and it's not completely correct, anyway, because there are non-terrestrial mammals and the KL divergence applies to any two probability distributions). $\endgroup$
    – whuber
    Dec 5, 2023 at 14:40
  • $\begingroup$ all fair comments. more content input that should be very useful $\endgroup$ Dec 6, 2023 at 1:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.