# If the KL divergence is not a metric or a measure, what is it?

The KL divergence is not a metric because e.g. it does not satisfy the symmetry property that metrics posses. According to the definition of measure, the KL divergence doesn't seem to be a measure, although the related Wikipedia article introduces the KL divergence as "a measure of how one probability distribution is different from a second". So, mathematically, what is the KL divergence?

Note that I know how the KL divergence is defined, so don't tell me how it's defined.

• It is a measure in the English sense. You can use the word the way the Wikipedia article does. You just have to avoid confusing measure theorists. :) May 13, 2020 at 0:49
• It is a (directed) divergence, see en.wikipedia.org/wiki/Divergence_(statistics) Dec 5, 2023 at 2:12
• (Divergence).. as mentioned in 'the related Wikipedia article'. Dec 5, 2023 at 8:35

The KL divergence is a pre-metric, given that it satisfies the two properties of pre-metrics

1. $$d(x, y) \geq 0, \forall x, y$$
2. $$d(x, x) = 0, \forall x$$

So, given that any pre-metric induces a topology, so does the KL divergence.

The KL divergence is related to other metrics (such as the Fisher information metric) or information-theoretic concepts (such as information content, mutual information, Shannon entropy, conditional entropy, and cross-entropy). See also the related Wikipedia page.

It is a divergence, a function that maps two probability densities (or mass function), $$p, q$$ into $$[0, \infty]$$ with many special properties.

In particular, KL is a Bregman divergence! Let $$\Omega$$ be the set of probability densities over your sample space $$X$$. Choose any function $$F:\Omega\to \mathbb{R}$$ that is continuously differentiable and strictly convex, and the function $$D_F: \Omega^2 \to \mathbb{R}, \ \ D(p, q) = F(p)-F(q)-\langle \nabla F(q), p-q\rangle$$ is a Bregman divergence.

Bregman divergences have many great properties such as being non-negative, $$D_F(p, q) = 0$$ iff $$p=q$$, convex in the first argument, ... (see Wikipedia page).

In particular if you set $$F(p) = -\log(p)$$, the induced Bregman divergence is indeed KL divergence! Hence, all the properties on that Wikipedia link apply.

Lastly note that as you mentioned, KL divergence is not a metric b/c it doesn't satisfy the triangle inequality. It is not a measure b/c it does not act on sets. It is a divergence.

• ... and has the property that if the distributions are identical, it maps them into $0$. Dec 5, 2023 at 2:06
• While not wrong, this is not particularly helpful Dec 5, 2023 at 2:07
• Agreed. The basic issue is that plenty of such functions are not divergences. It's kind of like trying to define "mammal" as a "terrestrial life form:" you get a tiny bit of information but not much enlightenment (and it's not completely correct, anyway, because there are non-terrestrial mammals and the KL divergence applies to any two probability distributions).
– whuber
Dec 5, 2023 at 14:40
• all fair comments. more content input that should be very useful Dec 6, 2023 at 1:44