I wanted to ensure that I understand this correctly. So would be grateful if someone can comment on whether my reasoning is correct. To take my running example, my joint distribution is given as follows:

$$p(w, \lambda, \phi) = p(\lambda) p(w|\lambda) P(\phi) \mathcal{L}(w, \phi)$$

where $\mathcal{L}(w, \phi)$ is the likelihood term depending only on $w$ and $\phi$. I had some help today and it seems that if I am interested in the marginal distribution, say on $\lambda$, I only need to consider the terms which have some dependence on $\lambda$.

So, we can say something like for the posterior distribution:

$$P(\lambda|w) \propto p(\lambda) p(w|\lambda)$$

Similarly,

$$P(\phi|w) \propto p(\phi) \mathcal{L}(w, \phi)$$

So, if we could evaluate the RHS of these equations, we have the probability density functions i.e. unto the normalisation constant for $\lambda$ and $\phi$?

Now, if my joint distribution was something where we have a new dependence where $\lambda$ now also depends on $\tau$ :

$$p(w, \lambda, \phi, \tau) = p(\lambda) p(w|\lambda) p(\lambda|\tau) p(\phi) \mathcal{L}(w, \phi)$$

Then to evaluate the posterior on $\lambda$, I will need to condition on both $w$ and $\tau$ i.e.

$$P(\lambda|w, \tau) \propto p(\lambda) p(w|\lambda) p(\lambda|\tau)$$

So, for a given $w$ and $\tau$, we can evaluate $P(\lambda|w, \tau)$ upto the normalisation constant?

Is the above description(s) correct?

• Looks right to me. Commented Mar 13, 2014 at 6:55
• You can't have two distributions for $\lambda$ in the same expression, $p(\lambda)$ and $p(\lambda|\tau)$. Maybe you meant to write $p(\lambda|w,\tau)\propto p(\tau)p(\lambda|\tau)p(w|\lambda)$? Otherwise it seems correct. Commented Mar 14, 2014 at 10:36
• I did not think of the model correctly but I sort of understood the concept that I was trying to figure out eventually :-)
– Luca
Commented Mar 14, 2014 at 11:56

Just going to elaborate, in case it might be useful or another novice.

Supposing we are interested in the marginal distribution of $\phi$, we start by marginalising out $\lambda$:

\begin{align} P(w, \phi) &= \int P(w, \lambda, \phi) \: d\lambda \\ &= P(\phi) \prod_{i=1}^{N}P(y_i \vert w_i, \phi) \int P(w \vert \lambda)\:d\lambda \\ &= \mathcal{K} P(\phi) \prod_{i=1}^{N}P(y_i \vert w_i, \phi) \\ \end{align}

where $\mathcal{K}$ is a constant representing the integral term which does not depend on $\lambda$.

Now, from definition we have:

$$P(\lambda|w) = \frac{P(\lambda, w)}{P(w)}$$

which can be written as: $$P(\lambda|w) \propto P(\phi) \prod_{i=1}^{N}P(y_i \vert w_i, \phi)$$

as $P(w)$ is also a constant.