# Tag Info

For discrete random variables $P$ and $Q$, the KL-divergence is defined as $$D_{KL}(P || Q) = \sum_x P(x) \ln\frac{P(x)}{Q(x)}$$ So, as $Q \rightarrow P$, the ratio $P(x)/Q(x)$ approaches $1$ for all $x$ and the logarithm $\ln P(x)/Q(x)$ approaches zero. As probabilities are bounded to the range $[0, 1]$, each term in the sum, $P(x) \ln\frac{P(x)}{Q(x)}$ ...