I don't completely understand KL divergence yet, and how the breakdowns are completely different for similar looking distributions.

In this explanation it says: $$KL(q_\lambda(z|x) \Vert p(z|x)) = E_q[\log \space q_\lambda(z|x)] - E_q[\log \space p(x,z)] + \log \space p(x)$$

The two distributions in the comparison look similar, both stating the probability distribution of $z$ given $x$, but in the first case it's the cross entropy ($q$ relative to $q$?), in the second case we're subtracting the cross entropy of $p(x,z)$ relative to $q$ plus $\log \space p(x)$.

Could someone explain why this problem should be solved like this?

  • 2
    $\begingroup$ To what does the phrase "this problem" refer?? $\endgroup$
    – whuber
    Dec 14, 2020 at 21:35
  • 2
    $\begingroup$ There is no problem to solve. This identity follows from the decomposition of the conditional density $p(z|x)$ as$$p(z|x)=\dfrac{p(x,z)}{p(x)}$$ $\endgroup$
    – Xi'an
    Dec 15, 2020 at 5:47

1 Answer 1


By definition, we have that the $\text{KL}$ divergence of the probability distributions $q_\lambda(z|x)$ and $p(z|x)$ is given by: $$\text{KL}\left(q_\lambda(z| x)\,\Vert\, p(z|x)\right) = \mathbb{E}_q\left[\log(q_\lambda(z|x)) - \log(p(z|x)) \right]$$ So it represents the expected value of the logarithmic difference between the probability distributions $p(z|x)$ and $q_\lambda(z| x)$ w.r.t. the probabilities given by $q_\lambda(z| x)$. This becomes useful to see how much one probability distribution differs from the other.

Given this, if we first apply Bayes' theorem to $p(z|x)$ (already noted in the comments), then use the quotient rule for logarithms and finally take advantage of the linearity of expectations, we get: $$\begin{align} \text{KL}\left(q_\lambda(z| x)\,\Vert\, p(z|x)\right) &= \mathbb{E}_q\left[\log(q_\lambda(z|x)) - \log\left(\frac{p(z,x)}{p(x)}\right) \right] \\ \\ &= \mathbb{E}_q\left[\log(q_\lambda(z|x)) - \log(p(z,x)) + \log(p(x)) \right] \\ \\ &=\mathbb{E}_q\left[\log(q_\lambda(z|x)) - \log(p(z,x)) \right] + \mathbb{E}_q[\log(p(x))]\\ \\ \end{align}$$

Now, we have to realize that $\log(p(x))$ is a function of $x$, while $q(z|x)$ is a function of $z$ ($x$ is given/ constant). Knowing this we can calculate $\mathbb{E}_q[\log(p(x))]$ as follows (assuming continuous case):

$$\mathbb{E}_q[\log(p(x))] =\int q(z|x)\log(p(x)) dz =\log(p(x)) \int q(z|x) dz= \log(p(x))$$

This way, we have just arrived to the expression given by the authors: $$ \text{KL}\left(q_\lambda(z| x)\,\Vert\, p(z|x)\right)= \mathbb{E}_q\left[\log(q_\lambda(z|x)) - \log(p(z,x)) \right] + \log(p(x))$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.