# performing parameter updates on Gamma distribution

I have the following form for a joint distribution

$$P(w, \lambda, \phi \vert y) = P(\phi) \times P(w \vert \lambda) \times P(\lambda) \times \prod_{i=1}^{N}P(y_i \vert w_i, \phi, \lambda)$$

I have some approximation for $w$ and what I am trying to do is update the parameters for the distributions for $\lambda$ and $\phi$. To take advantage of the conjugate prior properties, I have the following setup.

$\lambda$ is modelled as a Gamma distribution where $a$ and $b$ are the scale and rate parameters of the distribution respectively.

$$P(\lambda; a, b) = \frac{b^a \lambda^{a-1} \exp{(-b\lambda)}}{\Gamma(a)}$$

The log-posterior on $\lambda$ can be then written as: $$\log P(\lambda; a, b) = (a-1)\log(\lambda) -b\lambda + const(\lambda)$$

$\phi$ which is the noise precision is also modelled as a Gamma distribution and the log posterior can be written similarly as:

$$\log P(\phi; s, t) = (s-1)\log(\phi) -t\phi + const(\phi)$$

The log transformed prior on $w$ (which depends on $\lambda$ is: $$\log P(w \vert \lambda) = \frac{1}{2}(\log(\lambda \Lambda) - w^T(\lambda \Lambda)w) + const (\lambda, w)$$

This is just a zero mean MVN with the precision being defined as $\lambda \Lambda$.

Finally, the log-likelihood term can be written as:

$$\log P(y \vert x, w, \phi) = \frac{N}{2}\log(\phi) -\frac{\phi}{2} \sum_{i=1}^{N}(y_i - t(x_i, w_i)) + const(w, \phi)$$

x and y are observed and t is a non-linear transformation parameterised by $w$, which is the parameter of interest.

So, what I wanted to do was for a given $w$ obtain the posterior marginal distribution for $\lambda$. Now my, log posterior is given as (ignoring constant terms for the moment):

$$\mathcal{L} = \log P(\lambda; a, b) + \log P(\phi; s, t) + \log P(w \vert \lambda) + \log P(y \vert x, w, \phi)$$

So, I thought that we can condition on $w$ treating it as a constant and integrate over $\phi$ and we should get the parameters for $\lambda$ as the posterior should also be Gamma distributed due to conjugate properties. So, the posterior density on $\lambda$ is: $$P(\lambda) = \mathcal{L}\:d\phi$$

Now, doing the integrating I get something complicated as:

$$\phi((a-1)\log\lambda - b\lambda) + (s-1)\phi(log \phi - 1) -\frac{t\phi^2}{2} + \phi (\frac{1}{2}(\log(\lambda \Lambda) - w^T(\lambda \Lambda)w)) + \frac{N}{2}\phi(\log\phi -1) -\frac{\phi^2}{2}(y-t(x, w))^T(y-t(x, w))$$

So basically it becomes something complicated and it is not clear to me at all how the posterior Gamma distribution can be inferred from it which makes me think I have made a mistake with my update strategy. Can someone comment whether updating like this should be feasible? i thought it should be as this is whole point of using conjugate priors but I am at a loss as to what I have done wrong. I am inclined to think that the problem lies in the step

$$P(\lambda) = \mathcal{L}\:d\phi$$

but I am not sure what is the main issue.

To marginalize a distribution with respect to some parameter, you must integrate the density (not its logarithm).

But that is probably not the main issue: In the first line of your question the likelihood appears to depend on $\lambda$, but in the expression for the log-likelihood it depends only on $x$, $w$, and $\phi$. If the latter is correct, conditioned on knowing $w$, only the terms $P(\lambda)$ and $P(w|\lambda)$ are relevant for inference on $\lambda$ (all other terms are constant wrt. $\lambda$). Since in that case $\lambda$ is conditionally independent of $\phi$ you don't actually need to marginalize $\phi$ at all. You can directly compute the conditional (posterior) for $\lambda$, which as you know is Gamma distributed.

• Thanks for your reply. Yes, you are right that the log-likelihood is only dependent on $x$, $w$ and $\phi$. So, just to ensure I understand correctly, if I know $w$, then all I need to do is compute the expression for $P(\lambda) \times P(w|\lambda)$ and the required expression for the update for $\lambda$ should present itself. So if my current estimate for $\lambda$ distribution is given by scale and rate params $s_0$ and $r_0$, then after doing this computation, the updates for $s$ and $r$ should be available by simply comparing the terms to the Gamma distribution form?
– Luca
Mar 12, 2014 at 14:46
• Yes, exactly. You only need to consider the terms that include $\lambda$ and compare with the Gamma distribution. Mar 12, 2014 at 15:46
• if I may ask one clarification point...In the example above, $P(w|\lambda) \times P(\lambda)$ is the expression to use to update $\lambda$. However, looking at rules of probability, this is equal to the joint distribution $P(w, \lambda)$. So I am kinda confused as to how this represents the posterior $P(\lambda)$.
– Luca
Mar 12, 2014 at 16:55
• Sorry...yet another brain fart... $P(\lambda|w) \propto P(w|\lambda)P(\lambda)$
– Luca
Mar 12, 2014 at 17:42

As evident in your "something complicated" line, the posterior for $\lambda$ is Gamma distributed (as ensured by conjugacy), but only when $w$ and $\phi$ are known. This is all that conjugate priors provide.

• Thanks for the reply Tom. So, for example to update $\lambda$ I would also need to provide an approximation for $\phi$. Would something like the following make sense: 1: Start with approximation for $w$, $\phi$. 2: Estimate $\lambda$ using the explicit update 3: Estimate $\phi$ using known $w$ and estimate of $\lambda$. 4: Approximate $w$ using EP or any other method using the estimated $\phi$ and $\lambda$. 5: Iterate.
– Luca
Mar 12, 2014 at 12:30