Variational methods are applied when we are interested in a probability distribution $P$ but only have a tractably computable unnormalized form $\tilde{P}$ of $P$. Knowing the partition function $Z = \sum_x \tilde{P}(X)$ is desirable because it would allow us to cheaply compute $P(x)=\tilde{P}(x)/Z$.

Pretty much every single text on variational inference for probabilistic graphical models (for example Jaakola's "Tutorial on Variational Approximation Methods") describes the method as computing a probability distribution $Q$ that maximizes:

$$J(Q) = \log Z - KL(Q || P)$$

where $Q$ is a tractable distribution (meaning, it is easy to compute $Q(x)$). Because the $KL$ distance is always non-negative, $J(Q)$ is always a lower bound of $\log Z$. Therefore, finding a $Q^*$ that maximizes $J(Q)$ (or, equivalently, minimizes $KL$ since $\log Z$ is a constant in $Q$) yields the closest lower bound $J(Q^*)$ of $\log Z$ within the family to which $Q$ belongs.

This is often announced to much fanfare as if it were an obvious and useful goal.

However, I don't find it so obvious that a closest lower bound of $\log Z$ is a desirable goal. Sure, it gives us a lower bound $Z'$ of $Z$, but what is it useful for? Using it to compute an approximation $P'$ to $P$ by defining $P'(x) = \tilde{P}/Z'$ is an immediate idea but it does not sound that great because we wouldn't even have $P'$ to be a true distribution since it is not even guaranteed to sum up to 1.

So, why is computing this lower bound a useful thing?

PS: Another possible explanation is that the lower bound is not really the goal, but $Q^*$ is the goal, taken to be the closest approximation to $P$ within a tractable family of distributions. Fair enough, but then the whole thing could be explained much more succinctly and directly as a minimization of $KL(Q || P)$ instead of doing that as a step to minimize $J(Q)$. $J(Q)$ wouldn't even have to be defined in order to describe the method as finding $Q^*$ the closest approximation to $P$.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.