# Interpreting mixture of Gaussians (Variational Inference)

I've recently stated reading about mixture models and variational inference in this excellent paper, but I'm having troubles dissecting the models described, and have a couple of questions. Please see picture provided:

So the point here is to find out which latent variables are most probable given our observations?

What I'm having trouble interpreting is equation 7 and 8. Can I regard equation 7 as the numerator in equation 2, where mu and c are z (the latent variables?), and equation 8 as the denominator in equation 2?

I'm also having troubles understanding how they've derived equation 7 and 8. For equation 7, Is it some chain rule of the joint? And in equation 8, are they simply just marginalizing out each xi, and taking the product of each sample assuming they're i.i.d?

Since my calculus isn't really up to par, I assume the key here is that they want to show that integral in equation 8 is intractable, thus the need for Variational Inference?

I also have a question regarding the hyper parameter for the Gaussian for the means, we assume it is set (to something) in this model right? And how would I choose this if I were to create a model on my own?

Okay I’ll try and answer your questions one at a time:

1. Depending on your model different aspects of the latent variables will be of interest, but in the general context our aim is to just evaluate the posterior distribution of the latent variables given our data. Depending on exactly what your latent variables represent yes you might then be interested in which ones are most probable.

2. Yes, you’re right here, equation 7 is the numerator in equation 2 and equation 8 is the denominator in equation 2.

3. Equation 7 is just a standard use of conditional probability, namely that $$P(A,B) = P(A|B)P(B)$$ but fully factorised for all the latent variables.

4. Equation 8 is just the law of total probability, integrating out all the latent variables.

5. Basically yes, I’m not sure how intractable the integral is in this case, but generally you’ll find that marginalising out the latent variables quickly gets computationally intractable especially as the dimension increases.

6. Here the means are latent variables and unknown. In general with Bayesian methods you can always keep going deeper placing priors on any of the parameters, in the end you have to decide on something but you’ll just have to try a few different things

• Sorry for this late reply but i'd really appreciate any help. in equation 7 we took the product of all the $x$'s because they are independent and identically distributed but why didn't we do the same thing with $\mu$? they are independent and identically distributed. something like $\prod_{i=1}^{k} \mu_{i}$? – floyd Sep 7 at 13:43

Can I regard equation 7 as the numerator in equation 2, where mu and c are z (the latent variables?)

Yes, because they said that $$z = \{\mu, c\}$$

and equation 8 as the denominator in equation 2?

Yes.

For equation 7, Is it some chain rule of the joint?

Yes.

And in equation 8, are they simply just marginalizing out each xi, and taking the product of each sample assuming they're i.i.d?

They are marginalizing the $$\mu$$ using the integral and $$c$$ with the sum. Not the $$x$$. The integral and sum are the tools to "sum over" all possible values of that thing - so that then it falls out of the $$p(\mu, c, x)$$ and it becomes just $$p(x)$$ :-)

For equation 7, Is it some chain rule of the joint? And in equation 8, are they simply just marginalizing out each xi, and taking the product of each sample assuming they're i.i.d?

Well, they say that $$\mu$$ is latent, so it is not given, it is unknown, and thus so is $$\sigma$$. And bayesian hierarchical models are so cool because you don't have to choose - the proper value of a hyperparameter will be found by the fitting procedure :-) Either MCMC, MAP or some other method. It will be much slower, but it will work. I've seen, in some particular models, that there can be ways to approximate the hyperparameters by some rule of thumb to speed up the whole process and getting almost the same result. But in general, you want to estimate the optimal hyperparameter value and bayesian models do allow that :-)

• Thanks a bunch for the answer! :^) Regarding my question "And in equation 8, are they simply just marginalizing out each xi, and taking the product of each sample assuming they're i.i.d?", what would the correct nomenclature be? We marginalize out mu and c to obtain the marginal distribution over x? Could I say marginalize over x? – NotoriousFunk Aug 9 at 21:11
• @NotoriousFunk, no, you can say that you marginalized over possible values of $\mu$ and $c$, but not $x$! Or you can say that you integrated out $\mu$ and $c$. – Curious Aug 9 at 21:38
• @Curious Sorry for this late reply but i'd really appreciate any help. In equation 7 we took the product of all the $x$'s because they are independent and identically distributed but why didn't we do the same thing with $\mu$? they are independent and identically distributed. something like $\prod_{i=1}^{k} \mu_{i}$? – floyd Sep 7 at 13:46
• @floyd You could do that, but there's no point. The term $p(\boldsymbol{\mu})$ (note the $\boldsymbol{\mu}$ is bold which suggests it is a vector) is the probability of the $\boldsymbol{\mu}$ vector as a whole and there's no need to break it down to the product of all $p(\mu_{i})$. – Curious Sep 7 at 14:27