I want to learn how Gibbs Sampling works and I am looking for a good basic to intermediate paper. I have a computer science background and basic statistic knowledge.
Anyone has read good material around? where did you learn it?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
I'd start with:
Abstract: Computer-intensive algorithms, such as the Gibbs sampler, have become increasingly popular statistical tools, both in applied and theoretical work. The properties of such algorithms, however, may sometimes not be obvious. Here we give a simple explanation of how and why the Gibbs sampler works. We analytically establish its properties in a simple case and provide insight for more complicated cases. There are also a number of examples.
The American Statistician is often a good source for short(ish) introductory articles that don't assume any prior knowledge of the topic, though they do assume you have the background in probability and statistics that could reasonably be expected of a member of the American Statistical Association.
One online article that really helped me understand Gibbs Sampling is Parameter estimation for text analysis by Gregor Heinrich. It's not a general Gibbs sampling tutorial but it discusses it in terms of latent dirichlet allocation, a fairly popular Bayesian model for document modeling. It goes into the math in fair detail.
One that goes into even more exhaustive mathematical detail is Gibbs Sampling for the Uninitiated. And I mean exhaustive in that it assumes you know some multivariate calculus and then lays out every step from that point. So while there's a lot of math, none of it is advanced.
I assume these will be more useful to you then something that derives more advanced results, such as those that prove why Gibbs sampling converges to the correct distribution. The references I point out don't prove this.
The book Monte Carlo Strategies in Scientific Computing is an excellent resource. It does address things in a mathematically rigorous way, but you can easily skip mathematical sections that don't interest you and still get tons of practical advice out of it. In particular, it does a nice job of tying together Metropolis-Hastings and Gibbs sampling, which is crucial. In most applications you'll need to draw from a posterior distribution using Gibbs sampling, and so knowing how it fits into the logic of Metropolis in general is helpful.