The KL divergence is not a metric because e.g. it does not satisfy the symmetry property that metrics posses. According to the definition of measure, the KL divergence doesn't seem to be a measure, although the related Wikipedia article introduces the KL divergence as "a measure of how one probability distribution is different from a second". So, mathematically, what is the KL divergence?

Note that I know how the KL divergence is defined, so don't tell me how it's defined.

  • 1
    $\begingroup$ It is a measure in the English sense. You can use the word the way the Wikipedia article does. You just have to avoid confusing measure theorists. :) $\endgroup$ – hobbs May 13 at 0:49
  • $\begingroup$ @hobbs You can, but we should avoid it, given the confusion that it can cause. Also, if it's a pre-metric, why not calling it as such? Just get used to this new terminology, if you were not familiar with it. $\endgroup$ – nbro May 13 at 2:10

The KL divergence is a pre-metric, given that it satisfies the two properties of pre-metrics

  1. $d(x, y) \geq 0, \forall x, y$
  2. $d(x, x) = 0, \forall x$

So, given that any pre-metric induces a topology, so does the KL divergence.

The KL divergence is related to other metrics (such as the Fisher information metric) or information-theoretic concepts (such as information content, mutual information, Shannon entropy, conditional entropy, and cross-entropy). See also the related Wikipedia page.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.