Is VAE based on mean-field assumption? According to the slides (section 6), mean-field variational inference (aka. MFVI) assumes the latent variables ($z=\{z_1,..,z_m\}$ are independent from each other, and on top of this assumption, we can derive the update rule for each latent variable distribution $p(z_i)$ using Lagrange multiplier, and then iteratively update all $p(z)$ using coordinate ascent algorithm.
Given this definition of MFVI, I wonder if VAE is also using mean-field assumption, because in VAE each data-point $x_i$ has its own latent variable $z_i$, and different data-points' latent variables are independent, say $(x_i, z_i)$ has nothing to do with $(x_j, z_j)$.
I'm not very sure if this understanding correct, since in VAE the latent variable $z_i$ is more like the local latent variable, whereas in the MFVI they are global?
 A: Mean-Field Variational Inference
In general mean-field variational inference, there is no distinction between local and global variables.  The main elements are:

*

*Generative Model.
$$
p(z,x)
\tag{1a}
$$
where the latent variables $z$ can be partitioned into $M$ components, $z = \{z_1, \dots, z_m \}$. Each $z_i$ may be multi-dimensional, but we typically assume it is a one-dimensional scalar.


*Variational Family. Given observed data $x$, we approximate $p(z\mid x)$ using mean-field variational inference by optimizing over all distributions $q$ that take the form

$$
q(z;x) = \prod_{i=1}^{M}q_{i}(z;x)
\tag{2a}
$$
where I have included $x$ to highlight the fact that the variational distribution depends on the observed data. Let $Q$ be the collection of all probability distributions $q$ over $z$ that factorize via mean-field as in Eq (2).


*Optimization Objective.
$$
\min_{ q \in Q} \mathrm{KL}(q(z;x) \mid\mid p(z|x)).
\tag{3a}
$$
Using the calculus of variations, it can be shown that the optimal mean-field distribution $q^*(z)$ satisfies
$$
\begin{align}
q^*_i(z_i;x) \propto \exp\left\{ \mathbb{E}_{\prod_{j=1,j\ne l}^M q^*(z_j;x)}\left[ \log p(z, x) \right] \right\} && (i=1,\dots,m).
\end{align}
$$
The above equation tells us how to compute the optimal marginals $q^*(z_i)$ in terms of all the other marginals. If we are lucky enough to be able to compute the above expectations analytically, then we can find $q^*$ numerically by using coordinate ascent.  See Bishop (2006) for convergence conditions of coordinate ascent.
Variational Auto-Encoder / Amortized Inference
Variational auto-encoders make more assumptions than the mean-field case stated above. There are two main specializations.

*

*Generative Model.
$$
p_\theta(z,x) = \prod_{i=1}^m p_\theta(z_i)p_\theta(x_i | z_i).
\tag{1b}
$$
In contrast to the general form in Eq (1a), in Eq (2a) we are treating $\theta$ as "global" parameters (i.e., not latent variables) and setting $z_i$ to be the "local" latent variables. Here, each $i = 1,\dots,m$ indexes a data item, and each $z_i$ is typically multi-dimensional. Note that we are treating $\theta$ in a "non-Bayesian" way.


*Variational Family.
$$
q(z;x,\phi) = \prod_{i=1}^mq(z_i; x_i, \phi)
\tag{2b}
$$
again factorizes (i.e., mean-field). However, the fundamental difference between Eq (2b) and Eq (2a) is that the factors $q(z_i; x_i, \phi)$ are all parameterized by a shared parameter $\phi$ (typically the weights of a neural network). Unlike the "parameter-free" Eq (2a), we do not specify a separate marginal $q_i$ for each $z_i$ $(i=1,\dots,m)$ but we instead learn the parameter $\phi$, which gives us the approximate distribution over all $z_1, \dots, z_m$ at once.


*Optimization Objective.
$$
\min_{\phi}\mathrm{KL}(q(z;x,\phi) \mid\mid p_\theta(z|x)).
\tag{3b}
$$
Eq (3b) is again over more restricted family than Eq (3a). We have assumed above that $\theta$ is known and fixed, but we typically also optimize the parameters using the objective
$$
\min_{\theta,\phi}\mathrm{KL}(q(z;x,\phi) \mid\mid p_\theta(z|x))
$$
which is also a special case of Eq (3b) where $\theta$ is a latent variable with a uniform (improper prior) and the marginal $q_\theta$ of $q$ is constrained to be a dirac delta.
