Suppose I want to calculate the relative entropy:

$$D(q||p)= \sum q(x)\log \frac{q(x)}{p(x)}$$

If, for some $x$, I have $q(x)=p(x)=0$, does the corresponding factor in the sum becomes $0$?

  • 1
    $\begingroup$ meta.math.stackexchange.com/questions/5020/… Please use math typesetting. $\endgroup$
    – Sycorax
    Aug 4, 2016 at 13:05
  • $\begingroup$ If $q(x)=0$, why would you even include that term in the sum? By definition, it has no chance of occurring. $\endgroup$
    – whuber
    Aug 4, 2016 at 13:15
  • $\begingroup$ That's true, but I'm actually computing the KL distance of a Bayesian network, and I need to include such parameters in the code for the package to work. $\endgroup$
    – mackbox
    Aug 4, 2016 at 13:28
  • $\begingroup$ Yes, but doesn't that consideration alone immediately tell you what the only possible correct answer can be? $\endgroup$
    – whuber
    Aug 4, 2016 at 14:40

1 Answer 1


Wikipedia has the answer. Yes, the corresponding term becomes $0$.

In more detail (from wiki): For discrete distributions,

The Kullback–Leibler divergence is defined only if $q(x)=0$ implies $p(x)=0$, for all $x$ (absolute continuity). Whenever $p(x)$ is zero the contribution of the x-th term is interpreted as zero because $\lim _{x\to 0}x\log(x)=0$

That is if both probabilities are $0$ you take the limit which, is $0$ for that term of the sum.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.