2
$\begingroup$

Like in contamination models, some distributions have discrete component. e.g. $p(x) := (1 - \epsilon) q(x) + \epsilon \delta_{x_0}(x)$

In these distributions, is there a way to justify a definition of KL divergence?

e.g. $D(p_1 || p_2) = D((1 - \epsilon_1) q_1(x) || (1 - \epsilon_2) q_2(x)) + \epsilon_1 \log \frac{\epsilon_1}{\epsilon_2}$

$\endgroup$

1 Answer 1

3
$\begingroup$

The generic Kullback-Leibler divergence$$\int_\mathcal{X}\log\left(\dfrac{\text{d}P}{\text{d}Q}\right)\text{d}P$$is only defined when $P$ is absolutely continuous with respect to $Q$. In the case both $P$ and $Q$ have an extra atom at $x_0$, this is well-defined: the density of $P$ against the measure $\lambda+\delta_0$ [sum of Lebesgue and Dirac at zero] is $$\frac{\text{d}P}{\text{d}(\lambda+\delta_0)}(x)=(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)$$ and the same for $Q$: $$\frac{\text{d}Q}{\text{d}(\lambda+\delta_0)}(x)=(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)$$ Hence \begin{align*}\int_\mathcal{X}&\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right)\left\{(1 - \epsilon_1) q_1(x)\text{d}x + \epsilon_1 \text{d}\delta_{x_0}(x)\right\}\\ &=(1 - \epsilon_1) \int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right) q_1(x)\text{d}x+\\ &\quad \epsilon_1\int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x) + \epsilon_1 \mathbb{I}_{x_0}(x)}{(1 - \epsilon_2) q_2(x) + \epsilon_2 \mathbb{I}_{x_0}(x)}\right) \text{d}\delta_{x_0}(x)\\ &=(1 - \epsilon_1) \int_\mathcal{X}\log\overbrace{\left(\dfrac{(1 - \epsilon_1) q_1(x)}{(1 - \epsilon_2) q_2(x)}\right)}^\text{no mass in 0 under Lebesgue} q_1(x)\text{d}x+\\ &\quad \epsilon_1\int_\mathcal{X}\log\underbrace{\left(\dfrac{\epsilon_1 \mathbb{I}_{x_0}(x)}{\epsilon_2 \mathbb{I}_{x_0}(x)}\right)}_\text{weight of 0 under Dirac} \text{d}\delta_{x_0}(x)\\ &=(1 - \epsilon_1)\int_\mathcal{X}\log\left(\dfrac{(1 - \epsilon_1) q_1(x)}{(1 - \epsilon_2) q_2(x)}\right) q_1(x)\text{d}x+\epsilon_1\log\left(\dfrac{\epsilon_1}{\epsilon_2}\right) \end{align*}

$\endgroup$
2
  • $\begingroup$ Thank you, but would you mind teaching me, step-by-step, what logic you used for each modification to the equation? How did you get rid of the $1_{x_0}(x)$ inside the logarithm? $\endgroup$
    – diadochos
    Feb 14, 2018 at 1:08
  • $\begingroup$ No, no, I'm still interested, of course. I'm sorry for the delayed reply. (I was waiting for a notification of update, but I guess I missed it. Thank you for mentioning me) . Thanks to you, finally I was able to understand a justification for this kind of a calculus. Thank you very much for the clear demonstration! $\endgroup$
    – diadochos
    Feb 19, 2018 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.