I want to learn how Gibbs Sampling works and I am looking for a good basic to intermediate paper. I have a computer science background and basic statistic knowledge.
Anyone has read good material around? where did you learn it?
$\begingroup$
$\endgroup$
3
-
2$\begingroup$ googling "Gibbs sampling" isn't a bad way to get a range of views on the subject. I think its a good way to start, because you tend to approach it with a "skeptical mind" - you can't take googles word for granted, so you need to find a range of views. Of course you may need a reputable source at a later stage when you try to implement. But starting with the "reputable source" is not always the best idea, because they may be quite attached to a particular way of doing something - i.e. they know the "right way" and "all others are wrong or ineffective". $\endgroup$– probabilityislogicMar 19, 2011 at 10:26
-
3$\begingroup$ (+1) Questions that are easily answered by Googling are usually not welcome, but this one IMO attempts to capitalize on the collective wisdom of a community in a way that the Google ranking cannot do. It would be interesting to see what sources people really found useful for learning this material. $\endgroup$– whuber ♦Mar 19, 2011 at 14:05
-
4$\begingroup$ That is the problem. Google returns way too many results and not all the papers or tutorials are clear enough. $\endgroup$– fabrizioMMar 19, 2011 at 17:04
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
I'd start with:
Casella, George; George, Edward I. (1992). "Explaining the Gibbs sampler". The American Statistician 46 (3): 167–174. (FREE PDF)
Abstract: Computer-intensive algorithms, such as the Gibbs sampler, have become increasingly popular statistical tools, both in applied and theoretical work. The properties of such algorithms, however, may sometimes not be obvious. Here we give a simple explanation of how and why the Gibbs sampler works. We analytically establish its properties in a simple case and provide insight for more complicated cases. There are also a number of examples.
The American Statistician is often a good source for short(ish) introductory articles that don't assume any prior knowledge of the topic, though they do assume you have the background in probability and statistics that could reasonably be expected of a member of the American Statistical Association.
$\begingroup$
$\endgroup$
0
One online article that really helped me understand Gibbs Sampling is Parameter estimation for text analysis by Gregor Heinrich. It's not a general Gibbs sampling tutorial but it discusses it in terms of latent dirichlet allocation, a fairly popular Bayesian model for document modeling. It goes into the math in fair detail.
One that goes into even more exhaustive mathematical detail is Gibbs Sampling for the Uninitiated. And I mean exhaustive in that it assumes you know some multivariate calculus and then lays out every step from that point. So while there's a lot of math, none of it is advanced.
I assume these will be more useful to you then something that derives more advanced results, such as those that prove why Gibbs sampling converges to the correct distribution. The references I point out don't prove this.
$\begingroup$
$\endgroup$
The book Monte Carlo Strategies in Scientific Computing is an excellent resource. It does address things in a mathematically rigorous way, but you can easily skip mathematical sections that don't interest you and still get tons of practical advice out of it. In particular, it does a nice job of tying together Metropolis-Hastings and Gibbs sampling, which is crucial. In most applications you'll need to draw from a posterior distribution using Gibbs sampling, and so knowing how it fits into the logic of Metropolis in general is helpful.
