12
$\begingroup$

In Gelman & Hill (2007)'s book (Data Analysis Using Regression and Multilevel/Hierarchical Models), the authors claim that including redundant mean parameters can help speed up MCMC.

The given example is a non-nested model of "flight simulator" (Eq 13.9):

$$ \begin{align} y_i &\sim N(\mu + \gamma_{j[i]} + \delta_{k[i]}, \sigma^2_y) \\ \gamma_j &\sim N(0, \sigma^2_\gamma) \\ \delta_k &\sim N(0, \sigma^2_\delta) \end{align} $$

They recommend a reparameterization, adding the mean parameters $\mu_\gamma$ and $\mu_\delta$ as follows:

$$ \begin{align} \gamma_j \sim N(\mu_\gamma, \sigma^2_\gamma) \\ \delta_k \sim N(\mu_\delta, \sigma^2_\delta) \end{align} $$

The only justification offered is that (p. 420):

It is possible for the simulations to get stuck in a configuration where the entire vector $\gamma$ (or $\delta$) is far from zero (even though they are assigned a distribution with mean 0). Ultimately, the simulations will converge to the correct distribution, but we do not want to have to wait.

How do the redundant mean parameters help with this problem?

It seems to me that the non-nested model is slow mainly because of $\gamma$ and $\delta$ are negatively correlated. (Indeed, if one goes up, the other has to go down, given that their sum is "fixed" by the data). Do the redundant mean parameters help with reducing the correlation between $\gamma$ and $\delta$, or something else entirely?

$\endgroup$
2
  • $\begingroup$ Are you looking for intuitive insight on this particular problem (e.g. whether it is the correlation $\gamma$ - $\delta$ or the correlations $\gamma$ - $\mu$ and $\delta$ - $\mu$), or are you looking for intuitive insight on the general problem (ie. the concept of hierarchical centering)? In the latter case, would you desire intuition that is close to a proof or intuition that is much more loose and shows more or less how it works? $\endgroup$ Jan 23, 2018 at 13:01
  • $\begingroup$ I'd like intuitive insight on the concept of hierarchical centering in general (since the particular case in the question is directly an application of hierarchical centering). The key point I want insight on is: why does hierarchical centering work if the variance at the group level is a considerable part of the total variance? The paper by Gelfand et al. proves this mathematically (i.e. derive the correlation and find its limiting behavior), but without any intuitive explanation. $\endgroup$
    – Heisenberg
    Jan 23, 2018 at 22:23

1 Answer 1

4
$\begingroup$

The correlation to be avoided is the one between $\mu$ and the $\gamma_j$ and $\delta_k$.

By replacing $\gamma_j$ and $\delta_k$ in the computational model with alternative parameters that center around $\mu$ the correlation is reduced.

See for a very clear description section 25.1 'What is hierarchical centering?' in the (freely available) book 'MCMC estimation in MLwiN' by William J. Browne and others. http://www.bristol.ac.uk/cmm/software/mlwin/download/manuals.html

$\endgroup$
4
  • $\begingroup$ Section 25.1 of the 'MCMC estimation MlwiN' does describe this "hierarchical centering" technique, but does not go into any details further than claiming that it works. Digging through its references, I found that the actual proof of this technique is presented in the article Efficient parametrizations for normal linear mixed models, by Gelfand et al, Biometrika vol 82 issue 3. $\endgroup$
    – Heisenberg
    Jan 22, 2018 at 18:48
  • $\begingroup$ The article above in turn makes use of properties of the normal distribution without explaining. I found proofs of those properties in Conjugate Bayesian analysis of the Gaussian distribution by Kevin Murphy. $\endgroup$
    – Heisenberg
    Jan 22, 2018 at 18:50
  • $\begingroup$ Unfortunately, I still have not seen an intuitive explanation of why this technique works. $\endgroup$
    – Heisenberg
    Jan 22, 2018 at 18:52
  • $\begingroup$ It's late but I think this paper might be what you are looking for $\endgroup$
    – baruuum
    Apr 28, 2019 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.