# KL divergence and probabilities of 0 for P(i)

Why do probabilities of 0 for $P(i)$ not affect the result of the KL Divergence equation? Regardless of what probabilities we have for $Q(i)$, the product is 0.

What are the benefits of this? Is there some other equation that does them into account?

$$KL(P,Q) = \sum_{i=1}^n \ln\left(\frac{P(i)}{Q(i)}\right)*P(i)$$

• How would you distinguish the probability $(0.5,0.5)$ from the probability $(0.5,0.5,0)$ ? – Stéphane Laurent Aug 9 '15 at 14:22

One interpretation of $KL(P,Q)$ when P and Q are defined over a discrete space comes from encoding.
When you generate data according to P and you want to compress that data, like you would in a .zip file, you would like to use $\log(P(i))$ ressources to represent symbol $i$. That would be the most efficient way of representing data generated according to $P$.
What happens when you code instead symbols as if they were generated by $Q$ instead ? You then use $\log(Q(i))$ ressources to represent symbol $i$, which is less efficient. The difference in efficiency is $KL(P,Q)$
So, coming back to your question, it makes sense that symbols that never appear $P(i) = 0$ in the sequence we're looking at, do not affect the value of $KL(P,Q)$. Since they never appear, the cost of encoding them is 0