Skip to main content
2 of 4
include Q in first sentence so search engine snippet results are more meaningful
Silverfish
  • 23.9k
  • 28
  • 105
  • 206

Why the extra terms in scipy's implementation of Kullback-Leibler divergence?

Why do some definitions of Kullback-Leibler divergence include extra terms $-p_i + q_i$? For example kl_div() (in the Python scipy.special module) seems to define the Kullback-Leibler divergence as $$ \sum_i p_i \ln\frac{p_i}{q_i} - p_i + q_i. $$

The documentation says:

The origin of this function is in convex programming; see [1] for details. This is why the function contains the extra terms over what might be expected from the Kullback-Leibler divergence.

I don't have the referenced book at hand. Can someone provide a short justification/motivation for the additional $-p_i + q_i$ terms?

Anti-closing note: This is not a question about software, but about the concept behind it.

Igor F.
  • 9.7k
  • 1
  • 30
  • 64