How to justify a KL divergence when a distribution contains continuous and discrete components Like in contamination models, some distributions have discrete component.
e.g.
$p(x) := (1 - \epsilon) q(x) + \epsilon \delta_{x_0}(x)$
In these distributions, is there a way to justify a definition of KL divergence?
e.g.
$D(p_1 || p_2) = D((1 - \epsilon_1) q_1(x) || (1 - \epsilon_2) q_2(x)) + \epsilon_1 \log \frac{\epsilon_1}{\epsilon_2}$
 A: The generic Kullback-Leibler divergence$$\int_\mathcal{X}\log\left(\dfrac{\text{d}P}{\text{d}Q}\right)\text{d}P$$is only defined when $P$ is absolutely continuous with respect to $Q$. In the case both $P$ and $Q$ have an extra atom at $x_0$, this is well-defined: the density of $P$ against the measure $\lambda+\delta_0$ [sum of Lebesgue and Dirac at zero] is
$$\frac{\text{d}P}{\text{d}(\lambda+\delta_0)}(x)=(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)$$
and the same for $Q$:
$$\frac{\text{d}Q}{\text{d}(\lambda+\delta_0)}(x)=(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)$$
Hence
\begin{align*}\int_\mathcal{X}&\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right)\left\{(1 - \epsilon_1) q_1(x)\text{d}x + \epsilon_1 \text{d}\delta_{x_0}(x)\right\}\\
&=(1 - \epsilon_1) \int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right) q_1(x)\text{d}x+\\
&\quad \epsilon_1\int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right)  \text{d}\delta_{x_0}(x)\\
&=(1 - \epsilon_1) \int_\mathcal{X}\log\overbrace{\left(\dfrac{(1 - \epsilon_1) q_1(x)}{(1 - \epsilon_2) q_2(x)}\right)}^\text{no mass in 0 under Lebesgue} q_1(x)\text{d}x+\\
&\quad \epsilon_1\int_\mathcal{X}\log\underbrace{\left(\dfrac{\epsilon_1 \mathbb{I}_{x_0}(x)}{\epsilon_2 \mathbb{I}_{x_0}(x)}\right)}_\text{weight of 0 under Dirac}  \text{d}\delta_{x_0}(x)\\
&=(1 - \epsilon_1)\int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x)}{(1 - \epsilon_2) q_2(x)}\right) q_1(x)\text{d}x+\epsilon_1\log\left(\dfrac{\epsilon_1}{\epsilon_2}\right)
\end{align*}
