EDIT: After doing some more research, it seems slice sampling could be the way to go. I've seen it mentioned a lot in the context of how to sample the univariate distributions required for Gibbs Sampling. Any thoughts/opinions on this? Any other algorithms that might work better?
This is a half statistical and half methodological question, so please keep an open mind as to any type of solution.
I've been using Gibbs Sampling a lot recently to run Bayesian models and to sample from their posterior predictive distributions. Typically, I've been using the conjugate distributions, e.g. a linear regression with Gaussian-distributed $\beta$ with a Gaussian prior, and a Gamma-distributed noise term $\epsilon_t$. But it occurred to me that part of the real value of MCMC is supposed to be that, unlike message-passing and variational algorithms, the distributions don't necessarily need to be conjugate. Theoretically, I should be able to use any distribution, and the Gibbs Sampling should still work.
So, once I have the distribution $P(\theta | D) = P(D | \theta)P(\theta)$, what I have previously done is work out the form of the distribution on paper, transform it into a distribution I know about and figure out its parameters so I can sample from it.
My questions are:
- In situations like this is it better to use regular Metropolis-Hastings rather than Gibbs Sampling?
- Do you know of a good library for doing this kind of symbolic manipulation, so I don't have to do it by hand?
- If a conjugate distribution can't be found, for example if I'm using some crazy distribution I invented myself, what algorithms can most efficiently sample from $P(\theta | D)$ despite the fact that I don't know its normalization constant?
My main interest is in #3, since it's sort of a catch-all solution -- I'm aware of the family of rejection sampling algorithms, but it seems crazy that while running a sampling algorithm I'd have to spawn yet another sampling algorithm. What is the most efficient way to go about this?
Thanks very much!