0
$\begingroup$

I am working through the following review paper: https://arxiv.org/pdf/1601.00670.pdf and would like to gain some insight into arriving at the formulae for joint density of latent and observed variables, and evidence.

The setup is as follows.

Consider a mixture of K unit-variance, univariate Gaussians with means $ \boldsymbol \mu = ( \mu_{1},...\mu_{k}) $ drawn independently from common prior $p(\mu_{k})$ which is also a Gaussian $\mathcal{N}(0,\sigma). $

In order to draw an observation $ x_{i} $ from this distribution, $c_{i}$ is drawn from a categorical distribution over $(1...K)$, the value of $c_{i}$ dictates which Gaussian to sample from. Finally, the corresponding Gaussian is sampled generating $ x_{i}$ The variance of the k Gaussians is considered to be a hyperparameter here.

The joint Density is given as $ p(\boldsymbol {\mu, c, x)} = p(\boldsymbol \mu) \prod_{i=1}^n p(c_{i})p(x_{i}|c_{i},\boldsymbol \mu)$

Why are we able to move $p(\boldsymbol \mu)$ outside of the product? My belief is that the joint probability may be represented as $ p(x_{i}|c_{i},\boldsymbol \mu) p(c_{i}, \boldsymbol \mu)$ Seeing as $p(c_{i})$ is independent of $p(\boldsymbol \mu) $ This reduces to $ p(x_{i}|c_{i},\boldsymbol \mu) p(c_{i}) p ( \boldsymbol \mu)$ and we may move $p(\boldsymbol \mu)$ outside of the product as we choose this at the begining of the experiment.

Secondly, I am not totally confident in the derivation of the following two equations.

$ p(\boldsymbol{x}) = \int p (\boldsymbol\mu) \prod_{i=1}^{n} \sum_{c_{i}} p(c_{i})p(x_{i}|c_{i},{\mu}) $

$p(\boldsymbol x) = \sum_{ \boldsymbol c} \boldsymbol c \int p(\boldsymbol\mu) \prod_{i=1}^n p(x_{i}|c_{i},\boldsymbol \mu) d\boldsymbol \mu$

I believe that the first represents the marginal probability of $ \boldsymbol x$ acheived by summing conditionals over combinations of $c_{i}$ followed by the integration over the different mean parameters for the K Gaussians which may be assigned by sampling from the prior. (I am unsure about this part).

The final equation appears to be the result of integrating the marginal probability of $ x_{i}$ over the number of trials, over all possible values for $\boldsymbol \mu$ followed by a summation over all possible combinations of assignments of Gaussians $ \boldsymbol c $

Is my understanding of the equations anywhere near correct, I am a relative beginner to Bayesian statistics.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.