Like in contamination models, some distributions have discrete component. e.g. $p(x) := (1 - \epsilon) q(x) + \epsilon \delta_{x_0}(x)$

In these distributions, is there a way to justify a definition of KL divergence?

e.g. $D(p_1 || p_2) = D((1 - \epsilon_1) q_1(x) || (1 - \epsilon_2) q_2(x)) + \epsilon_1 \log \frac{\epsilon_1}{\epsilon_2}$


The generic Kullback-Leibler divergence$$\int_\mathcal{X}\log\left(\dfrac{\text{d}P}{\text{d}Q}\right)\text{d}P$$is only defined when $P$ is absolutely continuous with respect to $Q$. In the case both $P$ and $Q$ have an extra atom at $x_0$, this is well-defined: the density of $P$ against the measure $\lambda+\delta_0$ [sum of Lebesgue and Dirac at zero] is $$\frac{\text{d}P}{\text{d}(\lambda+\delta_0)}(x)=(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)$$ and the same for $Q$: $$\frac{\text{d}Q}{\text{d}(\lambda+\delta_0)}(x)=(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)$$ Hence \begin{align*}\int_\mathcal{X}&\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right)\left\{(1 - \epsilon_1) q_1(x)\text{d}x + \epsilon_1 \text{d}\delta_{x_0}(x)\right\}\\ &=(1 - \epsilon_1) \int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right) q_1(x)\text{d}x+\\ &\quad \epsilon_1\int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right) \text{d}\delta_{x_0}(x)\\ &=(1 - \epsilon_1) \int_\mathcal{X}\log\overbrace{\left(\dfrac{(1 - \epsilon_1) q_1(x)}{(1 - \epsilon_2) q_2(x)}\right)}^\text{no mass in 0 under Lebesgue} q_1(x)\text{d}x+\\ &\quad \epsilon_1\int_\mathcal{X}\log\underbrace{\left(\dfrac{\epsilon_1 \mathbb{I}_{x_0}(x)}{\epsilon_2 \mathbb{I}_{x_0}(x)}\right)}_\text{weight of 0 under Dirac} \text{d}\delta_{x_0}(x)\\ &=(1 - \epsilon_1)\int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x)}{(1 - \epsilon_2) q_2(x)}\right) q_1(x)\text{d}x+\epsilon_1\log\left(\dfrac{\epsilon_1}{\epsilon_2}\right) \end{align*}

  • $\begingroup$ Thank you, but would you mind teaching me, step-by-step, what logic you used for each modification to the equation? How did you get rid of the $1_{x_0}(x)$ inside the logarithm? $\endgroup$ – diadochos Feb 14 '18 at 1:08
  • $\begingroup$ No, no, I'm still interested, of course. I'm sorry for the delayed reply. (I was waiting for a notification of update, but I guess I missed it. Thank you for mentioning me) . Thanks to you, finally I was able to understand a justification for this kind of a calculus. Thank you very much for the clear demonstration! $\endgroup$ – diadochos Feb 19 '18 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.