# KL divergence of models with both continuous and discrete variables

When $$x$$ is discrete, KL divergence is $$D_{KL}(P||Q)=\sum\limits_{x}P(x)\log \frac{P(x)}{Q(x)}$$, when $$x$$ is continuous, $$D_{KL}(P||Q)=\int\limits_{x}p(x)\log \frac{p(x)}{q(x)}dx$$. However, when the space of the random variable $$x$$ is defined on mixed continuous and discrete space, what would be the KL divergence?

For example, $$x=(r,a)$$, where $$r$$ is a continuous variable that follows Gaussian distribution, $$a$$ is a discrete variable that follows Bernoulli distribution. $$r$$ and $$a$$ are independent of each other.
Under $$P(x)$$, $$r\sim \mathcal{N}(\mu_1,\sigma^2)$$ and $$a\sim \text{Bernoulli} (\beta)$$, i.e., $$P(r,a)=\left\{\begin{matrix} \mathcal{N}(\mu_1,\sigma^2))\cdot\beta, \quad a = 1, \forall r\in R\\ \mathcal{N}(\mu_1,\sigma^2))\cdot(1-\beta), \quad a = 0, \forall r\in R \end{matrix}\right.$$

Under $$Q(x)$$, $$r\sim \mathcal{N}(\mu_2,\sigma^2)$$ and $$a\sim \text{Bernoulli} (1-\beta)$$, i.e., $$Q(r,a)=\left\{\begin{matrix} \mathcal{N}(\mu_2,\sigma^2))\cdot(1-\beta), \quad a = 1, \forall r\in R\\ \mathcal{N}(\mu_2,\sigma^2))\cdot \beta, \quad a = 0, \forall r\in R \end{matrix}\right.$$

What is the KL divergence of $$P$$ and $$Q$$. Thank you very much fo the help!

In all cases, the KL divergence $$D_{KL}(p \parallel q)$$ is defined as the expected value of $$\log \frac{p(x)}{q(x)}$$, where the expectation is taken with respect to $$p$$:

$$D_{KL}(p \parallel q) = E_{p(x)} \left[ \log \frac{p(x)}{q(x)} \right]$$

In the discrete case, this involves summation:

$$D_{KL}(p \parallel q) = \sum_x p(x) \log \frac{p(x)}{q(x)}$$

And in the continuous case it involves integration:

$$D_{KL}(p \parallel q) = \int \log \frac{p(x)}{q(x)} dx$$

You can see that the formulas for discrete and continuous distributions simply follow from the definition of expected value in each of these cases.

The mixed discrete-and-continuous case is no different--both summation and integration are involved, as this is how expected value is defined. For example, consider joint distributions $$p(x,y)$$ and $$q(x,y)$$ where $$X$$ takes values in a discrete set $$\mathcal{X}$$ and $$Y$$ takes values in a continuous set $$\mathcal{Y} \subseteq \mathbb{R}$$. Then the KL divergence is:

$$D_{KL}(p \parallel q) \ = \ \sum_{x \in \mathcal{X}} \int_\mathcal{Y} p(x,y) \log \frac{p(x,y)}{q(x,y)} dy$$