# KL-divergence between two categorical/multinomial distributions gives negative values?

If

$$P = [0,0.9,0,0.1]$$

$$Q = [0,1,0,0]$$

Then $$KL(P||Q) = 0 + \ln(0.9/1)\cdot0.9 + 0 + 0 = -0.094$$

This shouldn't be possible from the Gibbs inequality. What am I misunderstanding?

Let’s remove the two categories with probability $0$ in both distributions. Your example is $P = (0.9, 0.1)$ and $Q = (1,0)$.
The KL divergence is $KL(P||Q) = \sum_i p_i \log\left( {p_i \over q_i }\right)$. It is not $$0.9 \times \log\, 0.9 + 0$$ but $$0.9 \times \log\, 0.9 + 0.1 \times ( +\infty ) = + \infty.$$