Why do some definitions of Kullback-Leibler divergence include extra terms $-p_i + q_i$? For example kl_div()
(in the Python scipy.special
module) seems to define the Kullback-Leibler divergence as
$$
\sum_i p_i \ln\frac{p_i}{q_i} - p_i + q_i.
$$
The documentation says:
The origin of this function is in convex programming; see
[1]
for details. This is why the function contains the extra terms over what might be expected from the Kullback-Leibler divergence.
I don't have the referenced book at hand. Can someone provide a short justification/motivation for the additional $-p_i + q_i$ terms?
Anti-closing note: This is not a question about software, but about the concept behind it.