To my (very modest) understand of variational inference, one tries to approximate an unknown distribution $p$ by finding a distribution $q$ that optimises the following:

$$KL (p||q) = \sum\limits_{x} p(x)log \frac {p(x)}{q(x)}$$

Whenever I invest time into understanding variational inference I keep hitting this formula and can't help but feel like I'm missing the point. It seems like I need to know $p$ in order to calculate $KL(p||q)$. But the whole point was I did not know this distribution $p$.

It's this exact point that's been bugging me every time I try to read up something variational. What am I missing?


I'll add a few extra comments here as a result of the answer of @wij, I'll attempt to be more precise.

In the cases that I am interested in, it indeed seems perfectly reasonable to consider that the following holds;

$$p(\theta | D) = \frac{p(D|\theta)p(\theta)}{p(D)} \propto p(D|\theta)p(\theta)$$

In this case I could know what $p$ should proportionally look like because I will have made a model choice for $p(D|\theta)$ and $p(\theta)$. Would I then be correct in saying that I then need to pick a family distribution $q$ [lets say gaussian] such that now I can estimate $KL(p(\theta|D) || q)$. It feels like in this case I am trying to fit a gaussian that is close to the non-normalized $p(D|\theta)p(\theta)$. Is this correct?

If so, it feels like I am assuming that my posterior is a normal distribution and I merely try to find likely values for this distribution with regards to the $KL$ divergence.


1 Answer 1


I have a feeling that you treat $p$ as a completely unknown object. I do not think this is the case. This is probably what you missed.

Say we observe $Y = \{y_i\}_{i=1}^n$ (i.i.d.) and we want to infer $p(x|Y)$ where we assume that $p(y|x)$ and $p(x)$ for $x\in\mathbb{R}^d$ are specified by the model. By Bayes' rule,

$$p(x|Y) = \frac{p(x)}{p(Y)}p(Y|x) = \frac{p(x)}{p(Y)}\prod_{i=1}^n p(y_i|x).$$

The first observation is that we know something about the posterior distribution $p(x|Y)$. It is given as above. Typically, we just do not know its normalizer $p(Y)$. If the likelihood $p(y|x)$ is very complicated, then we end up having some complicated distribution $p(x|Y)$.

The second thing that makes it possible to do variational inference is that there is a constraint on the form that $q$ can take. Without any constraint, $\arg \min_q KL(p||q)$ would be $p$ which is usually intractable. Typically, $q$ is assumed to live in a chosen subset of the exponential family. For example, this might be the family of fully factorized Gaussian distributions i.e., $q \in \mathcal{Q} = \{\prod_{i=1}^d q_i(x_i) \mid \text{each } q_i \text{ is a one-dimensional Gaussian}\}$. It turns out that if this is your constraint set, then each component of $q$ is given by

$$q_i \propto \exp( \mathbb{E}_{\prod_{j\neq i} q_j} \log p(x, Y) ), $$

where $p(x, Y) = p(x) \prod_{i=1}^n p(y_i|x).$ The exact formula does not matter much. The point is the approximate $q$ can be found by relying on the knowledge of the true $p$, and the assumption on the form that the approximate $q$ should take.


The following is to answer the updated part in the question. I just realized that I have been thinking about $KL(q||p(x|Y))$. I will always use $p$ for the true quantity, and $q$ for an approximate one. In variational inference or variational Bayes, $q$ is given by

$$q = \arg \min_{q \in \mathcal{Q}} KL(q\, ||\, p(x|Y)).$$

With the constraint set $\mathcal{Q}$ as above, the solution is the one given previously. Now if you are thinking about

$$q = \arg \min_{q \in \mathcal{Q}} KL( p(x|Y) \, || \, q),$$

for $\mathcal{Q}$ defined to be a subset of the exponential family, then this inference is called expectation propagation (EP). The solution for $q$ in this case is the one such that its moments match that of $p(x|Y)$.

Either way, you are right in saying that essentially you try to approximate the true posterior distribution in the KL sense by a distribution $q$ constrained to take some form.

  • $\begingroup$ I can't argue with this. I think most explanations including my own gloss over this. $\endgroup$ May 26, 2017 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.