7
$\begingroup$

In the Stan user's manual (Version 2.0.1, page 157), it says

A hierarchical model such as the above will suffer from the same kind of inefficiencies... [for a Hamiltonian Monte Carlo method] because the values of beta, mu_beta and sigma_beta are highly correlated in the posterior

They then recommend reparameterizing the model to avoid this issue.

I'm not sure I have good intuition about why correlations would be a problem. I thought that the whole point of Hamiltonian Monte Carlo was that it was invariant to rotation. So it shouldn't matter whether the posterior is aligned with the axes or if correlations put the major axis at an angle.

The best guess I can come up with is that, if the correlations get to be extreme enough, the posterior could end up very long and narrow, and that something about the difference in scales causes problems. But I'm not sure that this is a reasonable interpretation.

$\endgroup$
  • $\begingroup$ If you come to edit your question at some point, you might like to fix the typo in the title. $\endgroup$ – Glen_b -Reinstate Monica Dec 9 '13 at 22:25
9
$\begingroup$

Hamiltonian Monte Carlo is often quoted as being rotation invariant, but what does that actually mean?

In theory Hamiltonian Monte Carlo (HMC) is independent of the coordinates chosen to represent your distribution, which actually means it is both rotation and scale invariant. The problem is that, in practice, we can't run HMC exactly and must instead use numerical integrators for approximation. It is the numerical integrators that introduce a sensitivity to correlations and scales -- the more isotropic the target distribution the better-behaved the integrator.

That said, the reparameterizations recommended for hierarchical models are actually a bit more subtle. The issue isn't correlations in the classical sense but rather correlations in the hierarchical sense. These introduce different but equally challenging problems for HMC which can be mediated by non-centered parameterizations that cleverly shift the correlations around. For details check out our recent preprint, http://arxiv.org/abs/1312.0906.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.