How to justify a KL divergence when a distribution contains continuous and discrete components

Like in contamination models, some distributions have discrete component. e.g. $p(x) := (1 - \epsilon) q(x) + \epsilon \delta_{x_0}(x)$

In these distributions, is there a way to justify a definition of KL divergence?

e.g. $D(p_1 || p_2) = D((1 - \epsilon_1) q_1(x) || (1 - \epsilon_2) q_2(x)) + \epsilon_1 \log \frac{\epsilon_1}{\epsilon_2}$

The generic Kullback-Leibler divergence$$\int_\mathcal{X}\log\left(\dfrac{\text{d}P}{\text{d}Q}\right)\text{d}P$$is only defined when $P$ is absolutely continuous with respect to $Q$. In the case both $P$ and $Q$ have an extra atom at $x_0$, this is well-defined: the density of $P$ against the measure $\lambda+\delta_0$ [sum of Lebesgue and Dirac at zero] is $$\frac{\text{d}P}{\text{d}(\lambda+\delta_0)}(x)=(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)$$ and the same for $Q$: $$\frac{\text{d}Q}{\text{d}(\lambda+\delta_0)}(x)=(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)$$ Hence \begin{align*}\int_\mathcal{X}&\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right)\left\{(1 - \epsilon_1) q_1(x)\text{d}x + \epsilon_1 \text{d}\delta_{x_0}(x)\right\}\\ &=(1 - \epsilon_1) \int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right) q_1(x)\text{d}x+\\ &\quad \epsilon_1\int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right) \text{d}\delta_{x_0}(x)\\ &=(1 - \epsilon_1) \int_\mathcal{X}\log\overbrace{\left(\dfrac{(1 - \epsilon_1) q_1(x)}{(1 - \epsilon_2) q_2(x)}\right)}^\text{no mass in 0 under Lebesgue} q_1(x)\text{d}x+\\ &\quad \epsilon_1\int_\mathcal{X}\log\underbrace{\left(\dfrac{\epsilon_1 \mathbb{I}_{x_0}(x)}{\epsilon_2 \mathbb{I}_{x_0}(x)}\right)}_\text{weight of 0 under Dirac} \text{d}\delta_{x_0}(x)\\ &=(1 - \epsilon_1)\int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x)}{(1 - \epsilon_2) q_2(x)}\right) q_1(x)\text{d}x+\epsilon_1\log\left(\dfrac{\epsilon_1}{\epsilon_2}\right) \end{align*}
• Thank you, but would you mind teaching me, step-by-step, what logic you used for each modification to the equation? How did you get rid of the $1_{x_0}(x)$ inside the logarithm? Feb 14, 2018 at 1:08