I am working through the following review paper: https://arxiv.org/pdf/1601.00670.pdf and would like to gain some insight into arriving at the formulae for joint density of latent and observed variables, and evidence.

The setup is as follows.

Consider a mixture of K unit-variance, univariate Gaussians with means $ \boldsymbol \mu = ( \mu_{1},...\mu_{k}) $ drawn independently from common prior $p(\mu_{k})$ which is also a Gaussian $\mathcal{N}(0,\sigma). $

In order to draw an observation $ x_{i} $ from this distribution, $c_{i}$ is drawn from a categorical distribution over $(1...K)$, the value of $c_{i}$ dictates which Gaussian to sample from. Finally, the corresponding Gaussian is sampled generating $ x_{i}$ The variance of the k Gaussians is considered to be a hyperparameter here.

The joint Density is given as $ p(\boldsymbol {\mu, c, x)} = p(\boldsymbol \mu) \prod_{i=1}^n p(c_{i})p(x_{i}|c_{i},\boldsymbol \mu)$

Why are we able to move $p(\boldsymbol \mu)$ outside of the product? My belief is that the joint probability may be represented as $ p(x_{i}|c_{i},\boldsymbol \mu) p(c_{i}, \boldsymbol \mu)$ Seeing as $p(c_{i})$ is independent of $p(\boldsymbol \mu) $ This reduces to $ p(x_{i}|c_{i},\boldsymbol \mu) p(c_{i}) p ( \boldsymbol \mu)$ and we may move $p(\boldsymbol \mu)$ outside of the product as we choose this at the begining of the experiment.

Secondly, I am not totally confident in the derivation of the following two equations.

$ p(\boldsymbol{x}) = \int p (\boldsymbol\mu) \prod_{i=1}^{n} \sum_{c_{i}} p(c_{i})p(x_{i}|c_{i},{\mu}) $

$p(\boldsymbol x) = \sum_{ \boldsymbol c} \boldsymbol c \int p(\boldsymbol\mu) \prod_{i=1}^n p(x_{i}|c_{i},\boldsymbol \mu) d\boldsymbol \mu$

I believe that the first represents the marginal probability of $ \boldsymbol x$ acheived by summing conditionals over combinations of $c_{i}$ followed by the integration over the different mean parameters for the K Gaussians which may be assigned by sampling from the prior. (I am unsure about this part).

The final equation appears to be the result of integrating the marginal probability of $ x_{i}$ over the number of trials, over all possible values for $\boldsymbol \mu$ followed by a summation over all possible combinations of assignments of Gaussians $ \boldsymbol c $

Is my understanding of the equations anywhere near correct, I am a relative beginner to Bayesian statistics.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.