# Why KL divergence close to zero when Q close to P?

I was understanding cross-entropy and ended up understanding KL divergence. I learnt Cross entropy is Entropy + KL Divergence:

H(P, Q) = H(P) + D_KL(P||Q)

Minimizing Cross-entropy means minimizing KL Divergence. I further read that minimizing KL divergence means we are trying to make Q close to P. But, I really wanted to know why this happens? I read from many sources that when Q close to P, DKL close to zero but I didn't find any proper justification for this. I wonder if somebody has better insights on this.

• KL divergence has a relationship to a distance distance, if P and Q are close means distance between them is getting closer to zero. Some useful answers here, relating KL to a metric: stats.stackexchange.com/q/1031 Nov 8 '21 at 17:40
• What is it that you don't understand? If P and Q are identical, Q doesn't diverge at all from P and the KL-divergence is accordingly zero. That's how it's designed. Or is it the math that you don't understand? Have you looked up the definition of KL-divergence? Nov 8 '21 at 17:43
• @IgorF, yeah I understand KL-divergence will be zero when Q ~ P but I wanted to know what exactly happens when Q approaches P as I have a feeling that KL divergence will also getting smaller and finally becomes zero when Q = P. Nov 8 '21 at 17:48

For discrete random variables $$P$$ and $$Q$$, the KL-divergence is defined as
$$D_{KL}(P || Q) = \sum_x P(x) \ln\frac{P(x)}{Q(x)}$$
So, as $$Q \rightarrow P$$, the ratio $$P(x)/Q(x)$$ approaches $$1$$ for all $$x$$ and the logarithm $$\ln P(x)/Q(x)$$ approaches zero. As probabilities are bounded to the range $$[0, 1]$$, each term in the sum, $$P(x) \ln\frac{P(x)}{Q(x)}$$ also approaches zero and, consequently, the whole sum also approaches zero.