I'm working through variational Bayesian methods at the moment, and I think I have a grasp of the bigger picture. Where I sometimes have trouble is with the exact details of how it can be implemented. Right now, this centrs on the idea of a mean field variational family. Specifically, Blei et al. say the following:

In this review we focus on the mean-field variational family, where the latent variables are mutually independent and each governed by a distinct factor in the variational density. A generic member of the mean-field variational family is

$$q(z) = \prod_ {j=1}^m q_j (z_j )$$

I'm afraid that I can't see how a distribution can be expressed as a product in this way without being reduced to a constant. Clearly, I'm missing something fundamental, but I seem to be going around in circles trying to google the answer.

Can anyone supply some intuition?


Loosely speaking, the mean field family defines a specific class of joint distributions. So $z$ here is actually a parameter vector of length m. That means that $q(z)$ describes a joint distribution over all of the individual z's, and can be written as

$$q(z) = q(z_1, z_2, \ldots, z_m)$$

We can use the chain rule to factorize this:

$$ = q(z_1)q(z_2|z_1)\ldots q(z_m|z_1, z_2, \ldots z_{m-1})$$

Now, for this joint distribution to be in the mean field family, we make a simplifying assumption and assume that all of the $z_i$s are independent from each other. I'll note here that this assumes that the $z_i$'s under the variational distributions are independent; the true joint $p(z_1, \ldots z_m)$ is almost certainly going to have some dependence among the variables. In this sense, we are trading off accuracy (throwing away all covariances) for some computational benefits.

Now, if we make that independence assumption, we can see that the joint reduces down to

$$q(z) = q(z_1)q(z_2)\ldots q(z_m) = \prod_{i=1}^m q(z_i)$$

Which is the form that the mean field family takes. As for your question about how this won't reduce to a constant, I'm not entirely sure what you mean. All of the $z_i$'s are random variables, so I don't see how this could become a constant.

  • 1
    $\begingroup$ This is really helpful and has clarified things immensely. What was catching me out was where all the marginal probabilities went; by explaining that this is an approximation that trades off accuracy for computability over the joint distribution makes it much more intuitive. Thanks indeed! $\endgroup$ – Lodore66 Feb 10 '19 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.