After asking a question about Gibbs sampling earlier, I have another one for you.

I have not been able to find laymen's background on this, the only referenced use I've found for this is in statistical papers (Roberts et al. 2003, Gelfand et al. 1995), to name a few.

In an apparent try to speed up the convergence of Gibbs sampling, we might try a re-paramatrization of the parameters of interest. This is in order to make them less dependent on one another, as I understand it. Because, the less dependent, the faster the convergence.

The two different parametrization have been termed 'ancillary' and 'sufficient', or alternatively 'non-centered' and 'centered'. These were introduced to me in a fairly specific example, and I would therefore very much appreciate it if someone can introduce these concepts more generally (i.e., how is an ancillary parametrization different from a sufficient one). In my case, the sufficient parametrization is faster - is the sufficient parametrization always ideal?

  • $\begingroup$ The major entry on this topic is found in the papers of Xiao-Li Meng and his co-authors. $\endgroup$ – Xi'an Sep 7 '15 at 15:07
  • $\begingroup$ I will check that out, thanks! As for this topic - would it be possible to give an example of Gibbs sampling, for instance for a bivariate normal like in the other topic, and give examples of the ancillary and sufficient parameterizations? $\endgroup$ – user1991 Sep 7 '15 at 16:30
  • $\begingroup$ I suggest you read XL's papers, they are very well-written and full of examples. $\endgroup$ – Xi'an Sep 7 '15 at 18:53

My answer is based on Yu and Meng (2011).

Suppose there is a parameter $\theta$ and we are interested in its posterior distribution $p(\theta|y)$ based on some data $y$. We are interested in augmentation's $\phi$ with joint posterior $p(\theta,\phi|y)$ such that $$ \int p(\theta,\phi|y) d\phi = p(\theta|y). $$ We say $\phi$ is sufficient for $\theta$ if $$ p(y|\theta,\phi) = p(y|\phi), $$ i.e. the data $y$ are conditionally independent of $\theta$ given $\phi$. We say $\phi$ is ancillary for $\theta$ if
$$ p(\phi|\theta) = p(\phi), $$ i.e. the prior for $\phi$ does not depend on $\theta$.

In Yu and Meng (2011), there is a particularly simple example with $$ Y \sim N(\theta,1+V) $$ where a sufficient augmentation works well if $V$ is large while an ancillary augmentation works well if $V$ is small. So, no the sufficient parameterization is not always ideal.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.