KL divergence of models with both continuous and discrete variables When $x$ is discrete, KL divergence is $D_{KL}(P||Q)=\sum\limits_{x}P(x)\log \frac{P(x)}{Q(x)}$, when $x$ is continuous, $D_{KL}(P||Q)=\int\limits_{x}p(x)\log \frac{p(x)}{q(x)}dx$. However, when the space of the random variable $x$ is defined on mixed continuous and discrete space, what would be the KL divergence?
For example, $x=(r,a)$, where $r$ is a continuous variable that follows Gaussian distribution, $a$ is a discrete variable that follows Bernoulli distribution. $r$ and $a$ are independent of each other.
Under $P(x)$, $r\sim \mathcal{N}(\mu_1,\sigma^2)$ and $a\sim \text{Bernoulli} (\beta)$, i.e., $$P(r,a)=\left\{\begin{matrix}
\mathcal{N}(\mu_1,\sigma^2))\cdot\beta, \quad a = 1, \forall r\in R\\ 
\mathcal{N}(\mu_1,\sigma^2))\cdot(1-\beta), \quad a = 0, \forall r\in R
\end{matrix}\right.$$
Under $Q(x)$, $r\sim \mathcal{N}(\mu_2,\sigma^2)$ and $a\sim \text{Bernoulli} (1-\beta)$, i.e., $$Q(r,a)=\left\{\begin{matrix}
\mathcal{N}(\mu_2,\sigma^2))\cdot(1-\beta), \quad a = 1, \forall r\in R\\ 
\mathcal{N}(\mu_2,\sigma^2))\cdot \beta, \quad a = 0, \forall r\in R
\end{matrix}\right.$$
What is the KL divergence of $P$ and $Q$. Thank you very much fo the help!
 A: In all cases, the KL divergence $D_{KL}(p \parallel q)$ is defined as the expected value of $\log \frac{p(x)}{q(x)}$, where the expectation is taken with respect to $p$:
$$D_{KL}(p \parallel q) = E_{p(x)} \left[ \log \frac{p(x)}{q(x)} \right]$$
In the discrete case, this involves summation:
$$D_{KL}(p \parallel q) = \sum_x p(x) \log \frac{p(x)}{q(x)}$$
And in the continuous case it involves integration:
$$D_{KL}(p \parallel q) = \int p(x) \log \frac{p(x)}{q(x)} dx$$
You can see that the formulas for discrete and continuous distributions simply follow from the definition of expected value in each of these cases.
The mixed discrete-and-continuous case is no different--both summation and integration are involved, as this is how expected value is defined. For example, consider joint distributions $p(x,y)$ and $q(x,y)$ where $X$ takes values in a discrete set $\mathcal{X}$ and $Y$ takes values in a continuous set $\mathcal{Y} \subseteq \mathbb{R}$. Then the KL divergence is:
$$D_{KL}(p \parallel q) \ = \ \sum_{x \in \mathcal{X}}
\int_\mathcal{Y} p(x,y) \log \frac{p(x,y)}{q(x,y)} dy$$
