3
$\begingroup$

If

$$P = [0,0.9,0,0.1]$$

$$Q = [0,1,0,0]$$

Then $$KL(P||Q) = 0 + \ln(0.9/1)\cdot0.9 + 0 + 0 = -0.094$$

This shouldn't be possible from the Gibbs inequality. What am I misunderstanding?

$\endgroup$
7
$\begingroup$

Let’s remove the two categories with probability $0$ in both distributions. Your example is $P = (0.9, 0.1)$ and $Q = (1,0)$.

The KL divergence is $KL(P||Q) = \sum_i p_i \log\left( {p_i \over q_i }\right)$. It is not $$ 0.9 \times \log\, 0.9 + 0 $$ but $$ 0.9 \times \log\, 0.9 + 0.1 \times ( +\infty ) = + \infty.$$

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ @Elvis If there are 0's in an array .. how are we supposed to handle that situation? $\endgroup$ – Animesh Pandey Oct 13 '13 at 3:34
  • $\begingroup$ If two discrete distributions with same support are both 0 at some point of this support, you can just remove this point, as I did above. $\endgroup$ – Elvis Oct 13 '13 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.