# Kullback–Leibler divergence when one measure is a sum of diracs

In the book "Deep Learning" of Goodfellow, Bengio and Courville, section 5.5 of maximum likelihood estimation they explain a relation between the maximization of likelihood and minimization of the K-L divergence.

My question is on the formal construction there.

The divergence $KL(p,q)$ between two arbitrary probability measures is possible only if $p$ is absolutely continuous with respect to $q$. See Kullback–Leibler divergence.

In the book they have an abstract probability measure characterizing the underlying model $p_{data}$, which I assume is considered to be absolutely continuous with the Lebesgue measure on some Euclidean space $\mathbb{R}^n$, then $p_{data}$ is actually the density function of that probability measure.

A sample of $m$ points from $p_{model}$ is generated: $\{x_1, x_2, x_3, \dots, x_m\}$.

From 5.58 to 5.59 they convert a quantity of the form $$\frac{1}{m} \sum_{i=1}^{m} f(x_i)$$ to $$E_{x\sim \hat{p}_{data}} \big[f(x) \big]$$ Which means that the "empirical distribution" $\hat{p}_{data}$ is defined by $$\hat{p}_{data} = \frac{1}{m} \sum_{i=1}^{m} \delta_{x_i}$$

The last measure is not absolutely continuous with respect to $p_{data}$, how are they able to compute the KL-divergence there? or what am I missing?

• Beginning of en.wikipedia.org/wiki/… – Mark L. Stone Jun 7 '18 at 22:29
• @MarkL.Stone: I think the issue here is that $p_{data}$ is assumed to be continuous. – Alex R. Jun 7 '18 at 22:36
• @MarkL.Stone what do you mean? the first definition there is used when both measures are discrete, but only one is discrete in this case. – Jorge E. Cardona Jun 8 '18 at 21:12
• @JorgeE.Cardona If my answer has helped, could you mark it as 'accepted'. – CATALUNA84 Jun 26 at 7:45
• @CATALUNA84 Thank you for your answer. My question was not on implementation, but on the fact that to compute KL(p,q) one needs p to be absolutely continuous with respect to q, but that is not the case in that section. – Jorge E. Cardona Jun 29 at 14:55