0
$\begingroup$

Why do probabilities of 0 for $P(i)$ not affect the result of the KL Divergence equation? Regardless of what probabilities we have for $Q(i)$, the product is 0.

What are the benefits of this? Is there some other equation that does them into account?

$$KL(P,Q) = \sum_{i=1}^n \ln\left(\frac{P(i)}{Q(i)}\right)*P(i)$$

$\endgroup$
1
  • $\begingroup$ How would you distinguish the probability $(0.5,0.5)$ from the probability $(0.5,0.5,0)$ ? $\endgroup$ – Stéphane Laurent Aug 9 '15 at 14:22
0
$\begingroup$

One interpretation of $KL(P,Q)$ when P and Q are defined over a discrete space comes from encoding.

When you generate data according to P and you want to compress that data, like you would in a .zip file, you would like to use $\log(P(i))$ ressources to represent symbol $i$. That would be the most efficient way of representing data generated according to $P$.

What happens when you code instead symbols as if they were generated by $Q$ instead ? You then use $\log(Q(i))$ ressources to represent symbol $i$, which is less efficient. The difference in efficiency is $KL(P,Q)$

So, coming back to your question, it makes sense that symbols that never appear $P(i) = 0$ in the sequence we're looking at, do not affect the value of $KL(P,Q)$. Since they never appear, the cost of encoding them is 0

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.