What's a mean field variational family?

Question

I'm working through variational Bayesian methods at the moment, and I think I have a grasp of the bigger picture. Where I sometimes have trouble is with the exact details of how it can be implemented. Right now, this centrs on the idea of a mean field variational family. Specifically, Blei et al. say the following:

In this review we focus on the mean-field variational family, where the latent variables are mutually independent and each governed by a distinct factor in the variational density. A generic member of the mean-field variational family is

$$q(z) = \prod_ {j=1}^m q_j (z_j )$$

I'm afraid that I can't see how a distribution can be expressed as a product in this way without being reduced to a constant. Clearly, I'm missing something fundamental, but I seem to be going around in circles trying to google the answer.

Can anyone supply some intuition?

Answer 1

Loosely speaking, the mean field family defines a specific class of joint distributions. So $z$ here is actually a parameter vector of length m. That means that $q(z)$ describes a joint distribution over all of the individual z's, and can be written as

$$q(z) = q(z_1, z_2, \ldots, z_m)$$

We can use the chain rule to factorize this:

$$ = q(z_1)q(z_2|z_1)\ldots q(z_m|z_1, z_2, \ldots z_{m-1})$$

Now, for this joint distribution to be in the mean field family, we make a simplifying assumption and assume that all of the $z_i$s are independent from each other. I'll note here that this assumes that the $z_i$'s under the variational distributions are independent; the true joint $p(z_1, \ldots z_m)$ is almost certainly going to have some dependence among the variables. In this sense, we are trading off accuracy (throwing away all covariances) for some computational benefits.

Now, if we make that independence assumption, we can see that the joint reduces down to

$$q(z) = q(z_1)q(z_2)\ldots q(z_m) = \prod_{i=1}^m q(z_i)$$

Which is the form that the mean field family takes. As for your question about how this won't reduce to a constant, I'm not entirely sure what you mean. All of the $z_i$'s are random variables, so I don't see how this could become a constant.

Answer 2

To elaborate on and give context to previous answers, we expand on the use of mean-field variational assumptions in machine learning. The question can be decomposed into two main parts: 1) The definition of the mean-field distribution, and 2) How the mean-field distribution is optimized in variational inference.

Part 1:

The goal of variational inference is to estimate a surrogate distribution $q_{\phi}(z)$ for latent variables $z_{i} \in z$ to approximate a posterior distribution $p(z \vert x)$ as a function of a collected dataset $x$. As mentioned, your question mainly focuses on the mean-field assumption that factorizes the joint distribution into a set of independent latent factors

\begin{equation} \begin{split} q_{\phi}(z) \: &= \: \prod_{i}^{\text{N}} \: {q_\phi}_{i} (z_{i}) \end{split} \end{equation}

where $\phi$ refers to a particular family of distribution. It is this distribution that is used in lieu of the true posterior for modeling purposes.

Part 2:

Your reference to how the distribution is expressed as a constant seems to refer to how the surrogate distribution as a whole, and consequently, each latent factor $q_{\phi_{j}}(z_{j})$ is optimized. To optimize latent factor $q_{\phi_{j}}(z_{j})$, mean-field variational inference assumes that all latents $i \not = j$ are set as constant. The optimal solution $q*$ for each latent distribution is derived to be

\begin{equation} \begin{split} q_{\phi_{j}} (z_{j}) \: \approx \: \text{exp} \big( \: \mathbb{E}_{i \ne j} \: \big\lbrack \: \log p(z_{j}, x \vert \: z_{\neg_{j}}) \: \big\rbrack \: \big) \end{split} \end{equation}

where we have defined the distribution

\begin{equation} \begin{split} \log ~\widetilde{p}(z_{j}, x) \: &= \: \mathbb{E}_{i \ne j} \: \big\lbrack \: \log p(z_{j}, x \vert z_{\neg_{j}}) \: \big\rbrack \: + \: \text{C} \end{split} \end{equation}

\begin{equation} \begin{split} \mathbb{E}_{i \ne j} \: \big\lbrack \: \log p(z_{j}, x \vert z_{\neg_{j}}) \: \big\rbrack \: &= \: \int \: \log p(z_{j}, x \vert z_{\neg_{j}}) \cdot \prod_{\substack{i=1 \\ i \ne j}}^{\text{N}} q_{\phi_{i}(z_{i})} \: dz_{i} \end{split} \end{equation}

given a proportionality constant $C$. In practice, the optimization of each factor $q_{\phi_{j}}(z_{j})$ for $j \in \lbrace 1,2, \ldots \text{N} \rbrace$ follows an iterative strategy, hold $q_{\phi_{i \ne j}}$ fixed and maximize $\mathcal{L}_{elbo}$ with respect to the $z_{j}$ latent variable.

The following excerpts are taken from my book on variational inference. Learn more on the topic by visiting https://www.thevariationalbook.com/