I have been trying to understand variational inference (in a Bayesian context) where we’re trying to approximate $p(Z|X)$ where $Z$ is the set of latent variables and $X$ is the set of observable variables/data points, and I am not exactly sure what $Z$ means.

  • In Bishop’s book (PRML Chp10, p463), $Z=(z_1,…,z_n)$ is the same size as the observable variable set $X=(x_1,…,x_n)$, suggesting one latent variable is associated with one observable variable; but Bishop also says that the parameters are absorbed in Z. (in what way exactly?)
  • In Blei’s tutorial (https://arxiv.org/pdf/1601.00670.pdf), $X_{1:n}$ and $Z_{1:m}$ don’t have the same dimensions, suggesting Z does not depend on data size.
  • I also found this answer by Murphy (https://www.quora.com/What-makes-model-parameters-vs-latent-variables). It says ‘there is no real difference between parameters and other kinds of latent variables’ and clarifies that parameters are fixed in size whereas latent variables grow with data points (1 per data point). But it then proceeds to explain the relationships between parameter, latent var., and observable var. with these arrows I don’t understand. I think a distinction between the generative and discriminative case was made.
  • $\begingroup$ All this shows that the latent variable construction will depend on the model under study rather than being one-size-fits-all. $\endgroup$
    – Xi'an
    Commented Mar 4, 2021 at 19:18

1 Answer 1


Your question raises some important points which I also struggled with during self-study. I am by no means an expert, but I will attempt to clarify the various usages of the term "parameter" and "latent variable" and the ways in which the distinctions can get sharpened or blurred. Most of the time it really is a matter of context. There are a number of issues at play here in my opinion, and I understand the crux of each as follows:

The distinction, or lack thereof, between a parameter and a latent variable.

At the most general level, in Bayesian statistics, there is no difference between a parameter and a latent variable. Both are treated as unobserved random variables which we would like to compute a posterior distribution on given observed random variables (i.e. data).

This is relevant to variational inference because the nature of the method is inherently Bayesian. In the sense that it attempts to address situations where computing a posterior over unobserved random variables of interest given observed data is not tractable (for analytical or computational reasons). Its solution is to provide a method for computing a deterministic approximation to this intractable posterior, under arguably strong assumptions about how the family of approximations from which the specific approximation is selected factorises (the mean-field variational family).

Which now brings us to the arrows you stated that you did not understand in the 3rd link. This is a pictorial representation of a joint probability distribution, known as a directed graphical model/directed acyclic graph/Bayesian network and they are used to represent conditional independence relationships between random variables, which are represented as nodes (Murphy has not drawn the nodes because I guess it is not practicable to do so). This implies a particular factorisation of the postulated model/joint probability distribution, and the pictorial representations tend to assume a Bayesian statistical setting, although they are not limited to only Bayesian settings.

Using this representation, posterior inference amounts to computing a posterior on (possibly a subset of) the unobserved random variables, the unshaded nodes, using measurements of the observed random variables, the shaded nodes.

Returning to the variational inference setting, here is the Bayesian mixture of Gaussians model from Bishop's PRML p475. Note the unshaded nodes and unshaded nodes:

enter image description here

In this case, we cannot compute the following posterior over all unshaded nodes (unobserved random variables comprising the latent variables $\mathbf{Z}$ and parameters $\boldsymbol{\pi} ,\boldsymbol{\mu}, \boldsymbol{\Lambda}$) given the shaded nodes (observed data $\mathbf{X}$):

$$p(\mathbf{Z}, \boldsymbol{\pi} ,\boldsymbol{\mu}, \boldsymbol{\Lambda} | \mathbf{X}) = p(\boldsymbol{\pi}) p(\boldsymbol{\Lambda}) p(\boldsymbol{\pi} | \boldsymbol{\Lambda}) \prod^N_{n=1} p(\mathbf{z}_n | \boldsymbol{\pi}) p(\mathbf{x}_n | \mathbf{z}_n, \boldsymbol{\mu}, \boldsymbol{\Lambda})$$

Variational inference provides a means of approximating the above.

That latent variables and parameters are both unobserved random variables, and that observed data are observed random variables, is the sense in which Bishop means (p475 PRML):

From the perspective of graphical models, however, there is really no fundamental difference between them [i.e. the latent variables and parameters].

The informal characterisation of 'parameters' and 'latent variables' in terms of whether or not the quantities of each within a model increases as the number of observations in the data set grows.

Having blurred the distinction between a parameter and a latent variable above, sometimes authors will sharpen the distinction. For example on the same p475 of PRML, Bishop writes:

This example provides a nice illustration of the distinction between latent variables and parameters. Variables such as $\mathbf{z}_n$ that appear inside the plate are regarded as latent variables because the number of such variables grows with the size of the data set. By contrast, variables such as $\boldsymbol{\mu}$ that are outside the plate are fixed in number independently of the size of the data set, and so are regarded as parameters.

And in the Murphy quora comment you cited:

In practice the main difference is that parameters are usually fixed in number (independent of the size of your data), whereas latent variables are assumed to grow in number, since you usually have one per data case.

Very often in machine learning in particular, we may wish to introduce some additional structure into our probabilistic modelling, and we do this using latent variables. In this sense, a latent variable is an additional unobserved variable we introduce into the model for flexibility.

As you have seen in the Blei review paper and further to Xi'an's comment, there is in general no reason to expect there to be an association between the number of observations in a model, and the number of latent variables - ultimately it is for the context of the model and the responsibility of the modeller to specify this.

That being said, for many "conventional" models involving latent variables in machine learning (hidden Markov models, latent Dirichlet allocation, Bayesian mixture of Gaussians, the broader class of mixture models such as probablistic principal component analysis, sigmoid belief nets etc.), the additional flexibility and usefulness of these latent variable models derives from a natural association between the number of latent variables with the number of observations in the data set, so that as the latter grows in number, the former grows in number also.

Global variables (parameters) and local variables (latent variables) in variational inference.

As it is often the case that posterior inference in latent variable models cannot be exact, these models often require an approximate posterior inference method, of which variational inference is an instance of such method. Hence in the setting of variational inference, you may often observe a parity between the number of latent variables being used in the model, and the number of observations, as stated by Bishop and Murphy in the quotes above.

A different way of framing the distinction between random variables whose number are fixed in a model against those whose number are increasing with the size of the data set is the global-local distinction. This appears frequently appears in variational inference papers such as here and here. So parameters are characterised as global random variables - variables whose number do not increase with the data set size, and latent variables as local random variables, whose number do increase with the dataset size.

Observing the $n$-indexing on the local variables $z_{1:n}$, in contrast with the lack of indexing on the global variable $\beta$, these NeurIPs slides by Mohamed et al. show the general variational inference setup:

enter image description here

The difficulty in computing the posterior over the global variable (parameter) $\beta$ and the local variables (latent variables) $\mathbf{z} = z_{1:n}$, that is, $p(\beta, \mathbf{z} | \mathbf{x})$, is due to issues with computing the normalisation constant/evidence $p(\mathbf{x}) = \int \int p(\beta, \mathbf{z}, \mathbf{x}) d \beta \space d \mathbf{z}$. This is very often due to a problematic coupling between the global variable and the local variables.

Variational inference approximates this posterior by using the "best" distribution within a family of distributions referred to as the mean-field family:

enter image description here

This family is characterised by the fact that the dependency between the global variable $\beta$ and the local variables $\mathbf{z}$ is broken up, such that they are independent. That is, the main tractability assumption is that $q(\beta, \mathbf{z} | \lambda, \boldsymbol{\phi}) = q(\beta; \lambda) q(\mathbf{z}; \boldsymbol{\phi})$. The distinction between global and local variables can be seen in the fact that only $q(\mathbf{z}; \boldsymbol{\phi})$ further factorises into $\prod^n_{i=1} q(z_i; \phi_i)$.

Hopefully, I've adequately conveyed that the distinction between parameters and latent variables in terms of whether quantities of each increase or decrease with data set size can be a useful one in the context of variational inference, and that an alternative way of making this distinction is between global and local variables.

The differences in how distinctions between latent variables and parameters are treated in the specific cases of expectation-maximisation and variational inference.

You ask in "what way the parameter vector is absorbed into $Z$" and allude to the following quotation from Bishop's PRML p463:

This differs from our discussion of EM only in that the parameter vector $\boldsymbol{\theta}$ no longer appears, because the parameters are now stochastic variables and are absorbed into $\boldsymbol{Z}$.

Bishop is not only remarking on notation, but is alluding to a broader point about the difference in how variational inference and EM draw distinctions between latent variables and parameters in a frequentist-Bayesian sense.

That is, he is saying that in variational inference, we do not draw a distinction between a parameter and a latent variable - as we are in a Bayesian setting, they are both treated as unobserved random variables. In this case we are interested in approximating a posterior over all unobserved random variables of interest, the latent variables $\mathbf{Z}$ and the parameters $\boldsymbol{\theta}$, given observed data $\mathbf{X}$:

$$p(\mathbf{Z}, \boldsymbol{\theta} | \mathbf{X}) \approx q(\mathbf{Z}, \boldsymbol{\theta} | \boldsymbol{\phi}^*, \boldsymbol{\xi}^*)$$

Where $\boldsymbol{\phi}^*$ and $\boldsymbol{\xi}^*$ are the optimised variational parameters.

On the other hand, EM for maximum likelihood estimation in the presence of latent variables does make a distinction between a latent variable and a parameter in the sense that the latent variables $\mathbf{Z}$ are unobserved random variables, whilst the parameters $\boldsymbol{\theta}$ are fixed unknown numbers for which we want to compute point estimates $\hat{\boldsymbol{\theta}}$.

The situation muddies a little when we want to use EM for maximum a posteriori (MAP) estimation. In this case, the latent variables $\mathbf{Z}$ are still unobserved random variables, but the treatment of the parameters admits a Bayesian semantics - we speak of placing a "prior distribution $p(\boldsymbol{\theta})"$ on the parameters $\boldsymbol{\theta}$. However, the treatment of the parameters is still frequentist in the sense that at no point are we explicitly modelling uncertainty in the parameters $\boldsymbol{\theta}$, and we only confine ourselves to considering point estimates $\hat{\boldsymbol{\theta}}$, even if they are MAP point estimates. This post here gives a more complete treatment of the degree to which EM is "partially non-Bayesian".

To round this all off, this difference in how EM and variational inference treat latent variables and parameters, which Bishop is alluding to, is turned into an offhand, rather memorable claim about the "political philosophy of inference procedures" in Murphy's MLPP p753:

VBEM is also "egalitarian", since it treats parameters as "first class citizens", just like any other unknown quantity, whereas EM makes an artificial distinction between parameters and latent variables.

  • $\begingroup$ This is really helpful. Thank you. $\endgroup$
    – Nick
    Commented Mar 8, 2021 at 5:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.