Say I have a bunch of data from a Poisson distribution and I want to find out my posterior i.e. I'm data fitting:

$p(\lambda | X) \sim p(X|\lambda)p(\lambda)$

where $p(X|\lambda) = \frac{\exp(-\lambda)\lambda^x}{x!}$ so that my log-likelihood looks like:

$\log \mathcal{L}(\lambda|X) \sim x \log\lambda - \lambda$

Now as $\lambda > 0$, I transform my coordinates to be $\alpha = \log \lambda$. My new distribution looks like:

$p(X|\alpha) = \frac{\exp(-\exp(\alpha))\exp(\alpha x)}{x!}\cdot \bf{\exp(\alpha)}$

where the final $\exp(\alpha)$ comes from the Jacobian of the transformation.

This makes:

$\log \mathcal{L}(\lambda|X) \sim -\exp(\alpha) + \alpha x + \bf{\alpha}$

where the final $\bf{\alpha}$ in the new log-likelihood is from the earlier Jacobian.

The problem I'm having is that if I include that new $\alpha$ then my Metropolis-Hastings MCMC gives me a result that is incorrect. If I use a log-likehood that excludes it:

$\log \mathcal{L}(\lambda|X) \sim -\exp(\alpha) + \alpha x$

then I get correct results.

My question is: Why does the Metropolis-Hastings algorithm not care about the Jacobian?

  • $\begingroup$ $p(x\mid \alpha) = p(x\mid \exp(\lambda)) = p(x\mid f(\lambda))$ for any invertible $f$? $\endgroup$ – Hunaphu Jan 8 '15 at 0:11

You do not need the $\alpha$ since it is a parameter. The change of variables formula applies to the variable with respect to which you are "integrating". It is $x$ in your case. So MH is right to demand that you remove the excess factor.

So what you really have is:

$$ p(X|\alpha) = \frac{\exp(-\exp(\alpha))\exp(\alpha x)}{x!} $$

had you applied some transformation to your $x$ variable - then the change of variables foremula should be used.

EDIT To understand what's going on, think of a normal RV $X \sim \mathcal{N}(\mu ,\sigma^2)$. So $p(X|\mu,\sigma^2)$ is the density. If you transform $\mu$ with any transformation $f$, you get the new variable is $Y \sim \mathcal{N}(f(\mu) ,\sigma^2)$ and no jacobian is necessary. I hope you agree (if not, I'll have to write more in tex...).

If you want $\mathcal{P}(X\in A|\alpha)$ you'd integrate $x$ and keep $\alpha$ fixed - that's what I mean when I say "integrate". Probability is all about integration, after all.

So in the end you have $p(x|\alpha)$ with no extra jacobian term. Then proceed as usual with bayes' rule etc and you'll get the "right" density.

| cite | improve this answer | |
  • $\begingroup$ Why is it x that is being integrated over? Surely it is the unknown parameters when the full integral is written out? $\endgroup$ – Sigmund Fraud Jan 8 '15 at 7:58
  • $\begingroup$ I edited the answer above. $\endgroup$ – Yair Daon Jan 8 '15 at 13:30

Check your code, particularly the factors of N (number of data points) appearing in the likelihood. I find consistent results with the Jacobian factor (so the extra alpha) included in the log-posterior, and inconsistent inferences when I do not include the Jacobian (the opposite to what you say you are finding). The Jacobian comes in due to the prior, NOT the likelihood; in your analysis you implicitly assume a flat prior on lambda, so you need to translate this into a prior on alpha that contains the same assumptions (it will no longer be flat, of course) - the Jacobian factor does this translation for you.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.