I often see the mean-field approximation for Variational Bayes. I understand the independence assumption: what I don't understand is why we make that assumption. How does it help us?

  • 5
    $\begingroup$ The short answer is that it makes the minimization computationally tractable. Without these approximations, your posterior function could be anything at all, and with many data points, this is very difficult to calculate. Hence we approximate with a restricted class of distributions, usually which are assumed to be factorable over some partition of unobserved variables. $\endgroup$
    – Alex R.
    Apr 4 '16 at 22:35

To flesh the comment I write this answer.

Why do we use the mean-field approximation for variational Bayes?

Firstly we employ the variational Bayes to obtain the posterior distribution. But the posterior distribution $posterior = p(\theta|y) = \frac{likelihood * prior}{evidence} = \frac{p(y|\theta)*p(\theta)}{p(y)}$ is intractable for the evidence(also called marginal likelihood $\int p(y|\theta)d(\theta)$) is intractable(you cannot integrate over all possible $\theta$). Then to compute the posterior, we need to make the intractable problem tractable. The normal deterministic method is to approximate the marginal likelihood using a simpler distribution $q(\theta)$ which can be factorized to an integration of a sum of terms in the log joint $q(\theta)$ which is called mean field approximation. Absolutely this cannot be the only method you should use, since there are two other variational inference approaches: expectation propagation (Minka et al.) and variational 2nd-order Taylor approximation (Xing).

why we make that assumption. How does it help us?

Let $q$ be a member of a parametric family of density functions over a certain space(there are no dependencies between latent variables or a generalized version of this), then we can deform the marginal likelihood as:

$$\eqalign{ &\log p(y) &= \log p(y) \int q(\theta) d(\theta) = \int q(\theta) \log p(y) d\theta \\ & &= \int q(\theta) \log \{\frac{p(y, \theta)/q(\theta)}{p(\theta|y)/q(\theta)}\}d\theta\\ & &= \int q(\theta) log \{\frac{p(y, \theta)}{q(\theta)}\}d\theta + \int q(\theta) \log \{\frac{q(\theta)}{p(\theta|y)}\}d\theta \hspace{2cm} (1)\\ & &\ge \int q(\theta) \log \{\frac{p(y,\theta)}{q(\theta)}\}\hspace{7.3cm} (2) }$$

Let's denote the two terms in (1) respectively:

$\underline p(y;q) = exp\int q(\theta) log \{\frac{p(y, \theta)}{q(\theta)}\}d\theta$, which is the evidence lower bound, exp of (2)

$p_{KL} = exp\int q(\theta) \log \{\frac{q(\theta)}{p(\theta|y)}\}d\theta$, which is the exp of Kullback–Leibler divergence between $q$ and $p(·|y)$

Since the lower bound is more tractable than the evidence, we try to minimize the $p_{KL}$(which >= 0) by maximizing the lower bound. The lager the lower bound $\underline p(y;q)$ is, the more similar the $q(\theta)$ is to $p(\theta|y)$, and more accurately we get $\log p(y)$ and hence $p(y)$.

According to the mean field variational approximation, $q(\theta)$ factorizes into $\prod_{i=1}^M q_i(\theta_i)$, for some partition ${\theta_1,..., \theta_M}$ of $\theta$. We refer to each $q_i(\theta_i)$ as a local variational approximation.

If we keep $\mathcal{L}=\int q(\theta) \log \{\frac{q(\theta)}{p(\theta|y)}\}d\theta$, then $\mathcal{L}=\int \prod_{i=1}^M q_i(\theta_i)\{\log p(y, \theta)-\sum_{i=1}^M \log q_i(\theta_i)\}d\theta_1 \ldots d\theta_M$.

The do the coordinate ascend for each latent variable and keep all others fixed to compute $argmax_{q(\theta_i)}\mathcal{L}$.

May this be of help.

1) https://www.cs.cmu.edu/~epxing/Class/10708-17/notes-17/10708-scribe-lecture13.pdf
2) http://www.maths.usyd.edu.au/u/jormerod/JTOpapers/Ormerod10.pdf


Please read my recent paper titled "Gibbs sampler and coordinate ascent variational inference: A set-theoretical review". Visit link https://www.tandfonline.com/doi/full/10.1080/03610926.2021.1921214

My short answer is, by putting such a marginal independence via mean field variational assumption, we can use practical algorithm called coordinate ascent variational inference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.