When $x$ is discrete, KL divergence is $D_{KL}(P||Q)=\sum\limits_{x}P(x)\log \frac{P(x)}{Q(x)}$, when $x$ is continuous, $D_{KL}(P||Q)=\int\limits_{x}p(x)\log \frac{p(x)}{q(x)}dx$. However, when the space of the random variable $x$ is defined on mixed continuous and discrete space, what would be the KL divergence?

For example, $x=(r,a)$, where $r$ is a continuous variable that follows Gaussian distribution, $a$ is a discrete variable that follows Bernoulli distribution. $r$ and $a$ are independent of each other.
Under $P(x)$, $r\sim \mathcal{N}(\mu_1,\sigma^2)$ and $a\sim \text{Bernoulli} (\beta)$, i.e., $$P(r,a)=\left\{\begin{matrix} \mathcal{N}(\mu_1,\sigma^2))\cdot\beta, \quad a = 1, \forall r\in R\\ \mathcal{N}(\mu_1,\sigma^2))\cdot(1-\beta), \quad a = 0, \forall r\in R \end{matrix}\right.$$

Under $Q(x)$, $r\sim \mathcal{N}(\mu_2,\sigma^2)$ and $a\sim \text{Bernoulli} (1-\beta)$, i.e., $$Q(r,a)=\left\{\begin{matrix} \mathcal{N}(\mu_2,\sigma^2))\cdot(1-\beta), \quad a = 1, \forall r\in R\\ \mathcal{N}(\mu_2,\sigma^2))\cdot \beta, \quad a = 0, \forall r\in R \end{matrix}\right.$$

What is the KL divergence of $P$ and $Q$. Thank you very much fo the help!


1 Answer 1


In all cases, the KL divergence $D_{KL}(p \parallel q)$ is defined as the expected value of $\log \frac{p(x)}{q(x)}$, where the expectation is taken with respect to $p$:

$$D_{KL}(p \parallel q) = E_{p(x)} \left[ \log \frac{p(x)}{q(x)} \right]$$

In the discrete case, this involves summation:

$$D_{KL}(p \parallel q) = \sum_x p(x) \log \frac{p(x)}{q(x)}$$

And in the continuous case it involves integration:

$$D_{KL}(p \parallel q) = \int p(x) \log \frac{p(x)}{q(x)} dx$$

You can see that the formulas for discrete and continuous distributions simply follow from the definition of expected value in each of these cases.

The mixed discrete-and-continuous case is no different--both summation and integration are involved, as this is how expected value is defined. For example, consider joint distributions $p(x,y)$ and $q(x,y)$ where $X$ takes values in a discrete set $\mathcal{X}$ and $Y$ takes values in a continuous set $\mathcal{Y} \subseteq \mathbb{R}$. Then the KL divergence is:

$$D_{KL}(p \parallel q) \ = \ \sum_{x \in \mathcal{X}} \int_\mathcal{Y} p(x,y) \log \frac{p(x,y)}{q(x,y)} dy$$

  • $\begingroup$ Useful answer! :), as a sidenote I think there is a typo in the continuous case, wouldn't it be the following expression?: $$ D_{KL}(p\,\lVert\,q) =\int \color{blue}{p(x)}\log\left(\frac{p(x)}{q(x)}\right) dx$$ $\endgroup$
    – Javier TG
    Commented Jan 30, 2022 at 15:20
  • $\begingroup$ @JavierTG Yep, typo. Fixed it, thanks! $\endgroup$
    – user20160
    Commented Jan 30, 2022 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.