i am self-studying variational inference - and in Murphy's book "A probabilistic perspective on machine learning" it is discussed that minimizing the forward KL divergence (which is stated to be zero-avoiding for q) $$KL (p||q) = \sum\limits_{x} p(x)log \frac {p(x)}{q(x)}$$ gives different results than minimizing the reverse KL divergence $$KL (q||p) = \sum\limits_{x} q(x)log \frac {q(x)}{p(x)}$$ (which is stated to be zero-forcing for q). It is subsequently linked to the diagrams per below. I was hoping whehter someone can share the intuition on how the formulas for reverse and forward KL lead to the different approximations per below.

enter image description here


1 Answer 1


The KL divergences can be seen as a product of a weighting function $w(x)$ and a penalty function $g(x)$, i.e. $KL(q||p) = \sum_x w(x)g(x)$ with $w(x) = q(x)$ and $g(x) = \log\frac{q(x)}{p(x)}$ in the case of the reverse KL divergence. Whenever the weighting function is close to zero, i.e. $w(x) \approx 0$, the product of $w(x)g(x)$ is also close to zero and the value of the penalty function $g(x)$ does not contribute to the KL divergence no matter how large it is.

Consider first the case of reverse KL divergence. When $q(x)$ is close to zero, the penalty term $g(x) = \log\frac{q(x)}{p(x)}$ is ignored and therefore the reverse KL divergence ''ignores'' the portion of $p(x)$, which is not covered by $q(x)$. On the other hand, when $q(x)$ has significant mass and $p(x)$ is close to zero the penalty term $g(x)$ will be large. Therefore the reverse KL divergence discourages situations where $q(x)$ is high and $p(x)$ is small leading to the ''zero-forcing''-effect.

We can now make a similar analysis of the ''forward'' KL divergence. Now the weighting function corresponds to the target distribution $p$, i.e. $w(x) = p(x)$. Thus, when $p(x) \approx 0$ the value of the penalty $g(x) = \log\frac{p(x)}{q(x)}$ is largely ignored. Thus, there is almost no cost of having large $q(x)$ when $p(x)$ is small. On the other hand we see that when $p(x)$ has significant mass and $q(x)$ is small the contribution to the KL divergence is large. Combining these two properties leads to the ''zero-avoiding'' property.

  • 1
    $\begingroup$ super helpful. i now understand how hence in one case q(x) can cover parts of the space where p(x) has no density, and where in other cases, through the cost involved, the algo will avoid this. $\endgroup$
    – Wouter
    Commented Jul 29, 2015 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.