Suppose I want to calculate the relative entropy:
$$D(q||p)= \sum q(x)\log \frac{q(x)}{p(x)}$$
If, for some $x$, I have $q(x)=p(x)=0$, does the corresponding factor in the sum becomes $0$?
$\begingroup$
$\endgroup$
4
-
1$\begingroup$ meta.math.stackexchange.com/questions/5020/… Please use math typesetting. $\endgroup$– Sycorax ♦Aug 4, 2016 at 13:05
-
$\begingroup$ If $q(x)=0$, why would you even include that term in the sum? By definition, it has no chance of occurring. $\endgroup$– whuber ♦Aug 4, 2016 at 13:15
-
$\begingroup$ That's true, but I'm actually computing the KL distance of a Bayesian network, and I need to include such parameters in the code for the package to work. $\endgroup$– mackboxAug 4, 2016 at 13:28
-
$\begingroup$ Yes, but doesn't that consideration alone immediately tell you what the only possible correct answer can be? $\endgroup$– whuber ♦Aug 4, 2016 at 14:40
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Wikipedia has the answer. Yes, the corresponding term becomes $0$.
In more detail (from wiki): For discrete distributions,
The Kullback–Leibler divergence is defined only if $q(x)=0$ implies $p(x)=0$, for all $x$ (absolute continuity). Whenever $p(x)$ is zero the contribution of the x-th term is interpreted as zero because $\lim _{x\to 0}x\log(x)=0$
That is if both probabilities are $0$ you take the limit which, is $0$ for that term of the sum.
