3
$\begingroup$

I have the following form for a joint distribution

$$ P(w, \lambda, \phi \vert y) = P(\phi) \times P(w \vert \lambda) \times P(\lambda) \times \prod_{i=1}^{N}P(y_i \vert w_i, \phi, \lambda) $$

I have some approximation for $w$ and what I am trying to do is update the parameters for the distributions for $\lambda$ and $\phi$. To take advantage of the conjugate prior properties, I have the following setup.

$\lambda$ is modelled as a Gamma distribution where $a$ and $b$ are the scale and rate parameters of the distribution respectively.

$$ P(\lambda; a, b) = \frac{b^a \lambda^{a-1} \exp{(-b\lambda)}}{\Gamma(a)} $$

The log-posterior on $\lambda$ can be then written as: $$ \log P(\lambda; a, b) = (a-1)\log(\lambda) -b\lambda + const(\lambda) $$

$\phi$ which is the noise precision is also modelled as a Gamma distribution and the log posterior can be written similarly as:

$$ \log P(\phi; s, t) = (s-1)\log(\phi) -t\phi + const(\phi) $$

The log transformed prior on $w$ (which depends on $\lambda$ is: $$ \log P(w \vert \lambda) = \frac{1}{2}(\log(\lambda \Lambda) - w^T(\lambda \Lambda)w) + const (\lambda, w) $$

This is just a zero mean MVN with the precision being defined as $\lambda \Lambda$.

Finally, the log-likelihood term can be written as:

$$ \log P(y \vert x, w, \phi) = \frac{N}{2}\log(\phi) -\frac{\phi}{2} \sum_{i=1}^{N}(y_i - t(x_i, w_i)) + const(w, \phi) $$

x and y are observed and t is a non-linear transformation parameterised by $w$, which is the parameter of interest.

So, what I wanted to do was for a given $w$ obtain the posterior marginal distribution for $\lambda$. Now my, log posterior is given as (ignoring constant terms for the moment):

$$ \mathcal{L} = \log P(\lambda; a, b) + \log P(\phi; s, t) + \log P(w \vert \lambda) + \log P(y \vert x, w, \phi) $$

So, I thought that we can condition on $w$ treating it as a constant and integrate over $\phi$ and we should get the parameters for $\lambda$ as the posterior should also be Gamma distributed due to conjugate properties. So, the posterior density on $\lambda$ is: $$ P(\lambda) = \mathcal{L}\:d\phi $$

Now, doing the integrating I get something complicated as:

$$ \phi((a-1)\log\lambda - b\lambda) + (s-1)\phi(log \phi - 1) -\frac{t\phi^2}{2} + \phi (\frac{1}{2}(\log(\lambda \Lambda) - w^T(\lambda \Lambda)w)) + \frac{N}{2}\phi(\log\phi -1) -\frac{\phi^2}{2}(y-t(x, w))^T(y-t(x, w)) $$

So basically it becomes something complicated and it is not clear to me at all how the posterior Gamma distribution can be inferred from it which makes me think I have made a mistake with my update strategy. Can someone comment whether updating like this should be feasible? i thought it should be as this is whole point of using conjugate priors but I am at a loss as to what I have done wrong. I am inclined to think that the problem lies in the step

$$ P(\lambda) = \mathcal{L}\:d\phi $$

but I am not sure what is the main issue.

$\endgroup$
1
$\begingroup$

To marginalize a distribution with respect to some parameter, you must integrate the density (not its logarithm).

But that is probably not the main issue: In the first line of your question the likelihood appears to depend on $\lambda$, but in the expression for the log-likelihood it depends only on $x$, $w$, and $\phi$. If the latter is correct, conditioned on knowing $w$, only the terms $P(\lambda)$ and $P(w|\lambda)$ are relevant for inference on $\lambda$ (all other terms are constant wrt. $\lambda$). Since in that case $\lambda$ is conditionally independent of $\phi$ you don't actually need to marginalize $\phi$ at all. You can directly compute the conditional (posterior) for $\lambda$, which as you know is Gamma distributed.

$\endgroup$
  • $\begingroup$ Thanks for your reply. Yes, you are right that the log-likelihood is only dependent on $x$, $w$ and $\phi$. So, just to ensure I understand correctly, if I know $w$, then all I need to do is compute the expression for $P(\lambda) \times P(w|\lambda)$ and the required expression for the update for $\lambda$ should present itself. So if my current estimate for $\lambda$ distribution is given by scale and rate params $s_0$ and $r_0$, then after doing this computation, the updates for $s$ and $r$ should be available by simply comparing the terms to the Gamma distribution form? $\endgroup$ – Luca Mar 12 '14 at 14:46
  • 1
    $\begingroup$ Yes, exactly. You only need to consider the terms that include $\lambda$ and compare with the Gamma distribution. $\endgroup$ – Mikkel N. Schmidt Mar 12 '14 at 15:46
  • $\begingroup$ if I may ask one clarification point...In the example above, $P(w|\lambda) \times P(\lambda)$ is the expression to use to update $\lambda$. However, looking at rules of probability, this is equal to the joint distribution $P(w, \lambda)$. So I am kinda confused as to how this represents the posterior $P(\lambda)$. $\endgroup$ – Luca Mar 12 '14 at 16:55
  • $\begingroup$ Sorry...yet another brain fart... $P(\lambda|w) \propto P(w|\lambda)P(\lambda)$ $\endgroup$ – Luca Mar 12 '14 at 17:42
1
$\begingroup$

As evident in your "something complicated" line, the posterior for $\lambda$ is Gamma distributed (as ensured by conjugacy), but only when $w$ and $\phi$ are known. This is all that conjugate priors provide.

$\endgroup$
  • $\begingroup$ Thanks for the reply Tom. So, for example to update $\lambda$ I would also need to provide an approximation for $\phi$. Would something like the following make sense: 1: Start with approximation for $w$, $\phi$. 2: Estimate $\lambda$ using the explicit update 3: Estimate $\phi$ using known $w$ and estimate of $\lambda$. 4: Approximate $w$ using EP or any other method using the estimated $\phi$ and $\lambda$. 5: Iterate. $\endgroup$ – Luca Mar 12 '14 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.