Justification of independence assumption for latent variables in Expectation Maximization algorithm

Question

When deriving the ELBO/free energy in the EM algorithm, it is often done in a "general" case of observed and latent variables and then an assumption of independent (or iid) variables is brought in. For example, letting $\mathbf X$ be an observed variable and $\mathbf Z$ being unobserved/latent variable we can the ELBO/free energy as

\begin{align} \mathcal F\left(q(\mathbf Z), \boldsymbol \theta \right) & = \log p(\mathbf X \lvert \boldsymbol \theta) -\sum_{\mathbf{Z}} q\left(\mathbf{Z}\right) \log \frac{q\left(\mathbf{Z}\right)}{p\left(\mathbf{Z} \mid \mathbf{X}, \boldsymbol{\theta}\right)} \\ &= \log p(\mathbf X \lvert \boldsymbol \theta) -D_{\mathbb{K} \mathbb{L}}\left(q\left(\mathbf{Z}\right) \| p\left(\mathbf{Z} \mid \mathbf{X}, \boldsymbol{\theta}\right)\right) \\ &= \sum_{\mathbf Z} q(\mathbf z) \log \frac{p(\mathbf X, \mathbf Z \lvert \boldsymbol \theta)}{q(\mathbf Z)}. \end{align}

The argument continues that in (typical) cases where the observed variable $\mathbf X $ is composed of independent observations, it can be decomposed as $(\mathbf x_1, \ldots, \mathbf x_N)$ (e.g. a dataset of $N$ independent observations). Also, the latent variable can be decomposed as independent $(\mathbf z_1, \ldots, \mathbf z_N)$ but not justification is given for this latent variable independence (see [1], [2]). This is the step I don't see as necessarily obvious.

Under the assumption of both independence of data and of latent variables we can factor $p(\mathbf X, \mathbf Z) = \prod_{n=1}^N p(\mathbf x_n, \mathbf z_n)$ and it can be shown ([1], [2]) that the ELBO/free energy can be rewritten as \begin{align} \mathcal F\left(q(\mathbf Z), \boldsymbol \theta \right) & = \sum_{n=1}^N \sum_{\mathbf z_n} q_n(\mathbf z_n) \log \frac{p(\mathbf x_n, \mathbf z_n \lvert \boldsymbol \theta)}{q_n(\mathbf z_n)}. \end{align}

So, how can we justify that the latent variables are independent? Do the latent variables have to be independent for the observed variables to be independent? If so, why? What would happen if the latent variables were not independent?

---

References

[1]: pg.6; Neal, R.M., Hinton, G.E. (1998). A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants. Learning in Graphical Models. NATO ASI Series, vol 89. Springer, Dordrecht. https://doi.org/10.1007/978-94-011-5014-9_12

[2]: pg.453 Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg.

Answer 1

First, assume that the components of $X$ are independent, and that the components of $Z$ are independent. Bishop shows on page 453 that this means that the posterior also splits, i.e. $$ p(Z\mid X, \theta) = \prod_{n=1}^N p(z_n \mid x_n, \theta). $$

Then, since $q$ and $p(\cdot\mid x, \theta)$ both correspond to independent probability distributions, by the "tensorization" of KL divergence we have $$ D_{\text{KL}}(q(\cdot) \,\|\, p(\cdot \mid X, \theta))= \sum_{n=1}^N D_{\text{KL}}(q_n(\cdot) \,\|\, p_n(\cdot \mid x_n, \theta)). $$ This means $$ \log p(X\mid \theta) - D_{\text{KL}}(q(\cdot) \,\|\, p(\cdot \mid X, \theta)) = \sum_{n=1}^N \left[\log p(x_n \mid \theta) - D_{\text{KL}}(q_n(\cdot) \,\|\, p_n(\cdot \mid x_n, \theta))\right] $$

so under the assumptions of the $X$ being fully independent, and the $Z$ being fully independent, we arrive at the result that the joint ELBO is the sum of the marginal ELBOs. It is necessary for the latent variables to also be independent for this to work in general, otherwise we cannot factor $q$ and the posterior as we need.

Answer 2

As @jld has says, independence is important to the maths.

It's not always an assumption, however.

In some EM-algorithm settings the latent variables are real variables that we just didn't measure, but in other settings they are convenient fictions that let us build up models.

For example, the latent variables when phasing genotype data into haplotype data in genomics are real physical things that we would be able to observe with more expensive technology, but the latent variables in a typical variational autoencoder are just modelling fictions.

When the latent variables are real things, independence might be true or false. When independence is a modelling choice it's not something that can be true or false, only effective or ineffective.

Answer 3

Your question focuses primarily on the construction of the Bayesian model and the assumptions being made, rather than the specifics of the Expectation-Maximization (EM) algorithm.

Suppose we have a set of data points $x_{1}, x_{2},\ldots,x_{M}$. The EM method is an iterative optimization technique for maximizing the likelihood function where data may be incomplete and missing $x_{M+1}, x_{M+2}, \ldots, x_{M+n}$ or for latent variable models containing unobserved variables $z_{1}, z_{2},\ldots,z_{M}$.

The data points $x$ are typically assumed to be independent and identically distributed IID. This however does not have to hold, as in the case of structured approximations.

Likewise, for latent variable models, it is usually assumed each observation $x_{i}$ has a corresponding single latent variable $z_{i}$. This serves to simplify the graphical structure of the model, leading to an inherent tradeoff between model complexity and posterior expressiveness for capturing all possible dependencies.

A typical latent variable model diagram depicts an observation $x_{i}$, latent variable $z_{i}$ and model parameters $\Theta$. We note that the described derivations (in the answers above and in the Bishop text book) assume IID data and the mean-field approximation for the latent variables.

The following excerpts are taken from my book on variational inference. Learn more on the topic by visiting https://www.thevariationalbook.com/