performing parameter updates on Gamma distribution

Question

I have the following form for a joint distribution

$$ P(w, \lambda, \phi \vert y) = P(\phi) \times P(w \vert \lambda) \times P(\lambda) \times \prod_{i=1}^{N}P(y_i \vert w_i, \phi, \lambda) $$

I have some approximation for $w$ and what I am trying to do is update the parameters for the distributions for $\lambda$ and $\phi$. To take advantage of the conjugate prior properties, I have the following setup.

$\lambda$ is modelled as a Gamma distribution where $a$ and $b$ are the scale and rate parameters of the distribution respectively.

$$ P(\lambda; a, b) = \frac{b^a \lambda^{a-1} \exp{(-b\lambda)}}{\Gamma(a)} $$

The log-posterior on $\lambda$ can be then written as: $$ \log P(\lambda; a, b) = (a-1)\log(\lambda) -b\lambda + const(\lambda) $$

$\phi$ which is the noise precision is also modelled as a Gamma distribution and the log posterior can be written similarly as:

$$ \log P(\phi; s, t) = (s-1)\log(\phi) -t\phi + const(\phi) $$

The log transformed prior on $w$ (which depends on $\lambda$ is: $$ \log P(w \vert \lambda) = \frac{1}{2}(\log(\lambda \Lambda) - w^T(\lambda \Lambda)w) + const (\lambda, w) $$

This is just a zero mean MVN with the precision being defined as $\lambda \Lambda$.

Finally, the log-likelihood term can be written as:

$$ \log P(y \vert x, w, \phi) = \frac{N}{2}\log(\phi) -\frac{\phi}{2} \sum_{i=1}^{N}(y_i - t(x_i, w_i)) + const(w, \phi) $$

x and y are observed and t is a non-linear transformation parameterised by $w$, which is the parameter of interest.

So, what I wanted to do was for a given $w$ obtain the posterior marginal distribution for $\lambda$. Now my, log posterior is given as (ignoring constant terms for the moment):

$$ \mathcal{L} = \log P(\lambda; a, b) + \log P(\phi; s, t) + \log P(w \vert \lambda) + \log P(y \vert x, w, \phi) $$

So, I thought that we can condition on $w$ treating it as a constant and integrate over $\phi$ and we should get the parameters for $\lambda$ as the posterior should also be Gamma distributed due to conjugate properties. So, the posterior density on $\lambda$ is: $$ P(\lambda) = \mathcal{L}\:d\phi $$

Now, doing the integrating I get something complicated as:

$$ \phi((a-1)\log\lambda - b\lambda) + (s-1)\phi(log \phi - 1) -\frac{t\phi^2}{2} + \phi (\frac{1}{2}(\log(\lambda \Lambda) - w^T(\lambda \Lambda)w)) + \frac{N}{2}\phi(\log\phi -1) -\frac{\phi^2}{2}(y-t(x, w))^T(y-t(x, w)) $$

So basically it becomes something complicated and it is not clear to me at all how the posterior Gamma distribution can be inferred from it which makes me think I have made a mistake with my update strategy. Can someone comment whether updating like this should be feasible? i thought it should be as this is whole point of using conjugate priors but I am at a loss as to what I have done wrong. I am inclined to think that the problem lies in the step

$$ P(\lambda) = \mathcal{L}\:d\phi $$

but I am not sure what is the main issue.

Answer 1

To marginalize a distribution with respect to some parameter, you must integrate the density (not its logarithm).

But that is probably not the main issue: In the first line of your question the likelihood appears to depend on $\lambda$, but in the expression for the log-likelihood it depends only on $x$, $w$, and $\phi$. If the latter is correct, conditioned on knowing $w$, only the terms $P(\lambda)$ and $P(w|\lambda)$ are relevant for inference on $\lambda$ (all other terms are constant wrt. $\lambda$). Since in that case $\lambda$ is conditionally independent of $\phi$ you don't actually need to marginalize $\phi$ at all. You can directly compute the conditional (posterior) for $\lambda$, which as you know is Gamma distributed.

Answer 2

As evident in your "something complicated" line, the posterior for $\lambda$ is Gamma distributed (as ensured by conjugacy), but only when $w$ and $\phi$ are known. This is all that conjugate priors provide.