I am reading Elements of information theory by Thomas M. Cover, and Joy A. Thomas second edition. and on page 19, chapter 2, section 3, it says:

enter image description here

I am confused by the random mentioning of triangles.

Out of the blue, this sounds like a typo. I am starting to understand relative entropy and its dependence on probability mass distributions but I have not used any triangles in any of my proofs.

Why does the triangle inequality have anything to do with entropy?

I have copied the text from the image below:

In the above definition, we use the convention that $0\log \frac00 =0$ and the convention (basedoncontinuity arguments) that $0log \frac0q =0$ and $p\log \frac{p}0 =\infty$. Thus, if there is any symbol $x \in X$ such that $p(x) > 0$ and $q(x)=0$, then $D(p||q)=\infty$.

We will soon show that relative entropy is always nonnegative and is zero if and only if $p = q$. However, it is not a true distance between distributions since it is not symmetric and does not satisfy the triangle inequality. Nonetheless, it is often useful to think of relative entropy as a “distance” between distributions.


1 Answer 1


This is due to the definition of distance in the sense of metric space:

In mathematics, in particular geometry, a distance function on a given set $M$ is a function $d: M \times M \to \mathbb{R}$, where $\mathbb{R}$ denotes the set of real numbers, that satisfies the following conditions:

  • $d(x,y) \ge 0$, and $d(x,y) = 0$ if and only if $x = y$. (Distance is positive between two different points, and is zero precisely from a point to itself.)
  • It is symmetric: $d(x,y) = d(y,x)$. (The distance between $x$ and $y$ is the same in either direction.)
  • It satisfies the triangle inequality: $d(x,z) \le d(x,y) + d(y,z)$. (The distance between two points is the shortest distance along any path). Such a distance function is known as a metric. Together with the set, it makes up a metric space.

It just says that we do not have $D(p||q) \le D(p||r) + D(r||q)$ in general. In fact, it is not even symmetric.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.