Sampling from conditional distribution in general case

I'm dealing with Gibbs Sampling now. Let's consider the example: I know the distribution of X|Y and the distribution of Y. They are some known - Binomial or Beta or other but particular. Thus I have in analytical view f(X|Y), f(Y) and I can calculate joint distribution f(X,Y). To provide Gibbs Sampling I need to calculate many times X∼X|Y and Y∼Y|X. The question is: which technique can I use to sample Y∼Y|X in general case, for any given f(X|Y), f(Y)?

1 Answer

It is not exactly like that. If it is easy and effective to sample from $X\mid Y=y$ and $Y\mid X=x$, then plain vanilla Gibbs sampling is probably the way to go for sampling from the joint distribution of $X,Y$. There may be cases when it is difficult to sample from some of the full conditionals. In those cases, something like a "Metropolis within Gibbs" algorithm may be a good strategy. Take a look at these notes. A suggestion: I believe that you shouldn't study the subject this way. You must abstract the general concepts working with specific concrete examples. Trying to extend those concrete examples will show you where the difficulties arise. If you're looking for a very good book on the subject, check out the latest edition of Robert and Casella.

• But when I do something in OpenBugs, for example, I just need to write {T∼dbeta(3,7); X∼dbin(T,n)} n=15. And this program through Gibbs Sampling automatically sample many times X|T and T|X. It means that they use some numerical technique for sampling T|X cause I can put any distributions and I'll get correct results. How can I do this, sample from T|X, in R, for example? – Oleg Mar 26 '13 at 15:07
• We need to know more about OpenBUGS internals to answer that. From their site: "BUGS includes a range of algorithms that its expert system can assign to each such computational task." – Zen Mar 26 '13 at 17:43
• I agree that there is a lot it BUGS algorithms but I'm interesting just in sampling from conditional distributions. I don't think, they introduced something new. Perhaps they are using just known technique – Oleg Mar 27 '13 at 13:35
• Did you read the notes pointed in my answer? They are probably using "Metropolis within Gibbs", but I can't know that for sure. – Zen Mar 27 '13 at 17:50