I have fairly good practical experience with Metropolis-Hastings and Gibbs sampling, but I want to get a better mathematical understanding of these algorithms. What are some good textbooks or articles that prove the correctness of these samplers (more algorithms would also be great)?
I'm not sure whether this is exactly what you're after, but a couple of articles I've found useful on theoretical properties of various Metropolis-Hastings algorithms are:
Optimal scaling for various Metropolis-Hastings algorithms - Roberts & Rosenthal, 2001.
(This summarises some earlier results for the Ransom walk Metropolis and the Metropolis-adjusted Langevin algorithm.)
The Random Walk Metropolis: linking theory and practice through a case study - Sherlock, Fearnhead & Roberts, 2009
(This may be a good bridge between theoretical properties and practical use, as suggested by the title.)
The book by Robert & Casella (mentioned above) is a very good and thorough resource, but you may also find these two of use:
Markov Chain Monte Carlo in Practice - Gilks, Richardson & Spiegelhalter (1995)
Markov Chain Monte Carlo: Stochastic simulation for Bayesian inference - Gamerman (2006)
These both also have information on Gibbs sampling. I suppose you may also find some other information on Markov Chains useful. I generally use:
- Markov Chains - Norris, (1998).
But there is also some good information in most MCMC books on this.