Difference between KLdiv(P||Q) and KLdiv(Q||P) in variational inference Variational inference is about finding an estimation Q(z) for the posterior P(Z|x).
According to all the variational inference papers, this is done by minimizing the KLdiv(Q||P). I want to understand why they choose to minimize this KL div and not the KLdiv(P||Q)? and how this choice is beneficial for variational inference?
 A: Both divergences are legitimate. In fact, they are members of a continuous family of divergences called the $\alpha$-divergences, which all have their justifications.
If you look at the definition of the Kullback-Leibler divergence:
$$
\operatorname{KL}(Q\|P) = \int Q \log \frac{Q}{P}
$$
you see that the approximation $Q$ has to be zero whenever $P$ is zero, so this version is also sometimes called zero-forcing. This leads to approximations that underestimate the support $P$. E.g., approximating a Gaussian mixture $P$ with a single Gaussian $Q$ would lead to $Q$ choosing one of the components of the mixture and trying to approximate that one well while "ignoring" the others.
On the other hand, if you use $\operatorname{KL}(P\|Q)$, the same argument leads to a trend to overestimate the support of $P$. And the approximation of a Gaussian mixture $P$ with a single Gaussian $Q$ would lead to the Gaussian using a large variance to cover all the components of the mixture.
There is a nice visualization of the two above applications to Gaussian mixtures in Bishop p.469, and I tried to reproduce it here. In the image, you see on the left side the zero-forcing and on the right side the zero-avoiding case. The black contours belong to the Gaussian mixture to approximate and the red contours belong to the single Gaussian approximation.

In summary, both versions have their virtues and usage depends on what you need.
