I am learning Gibbs Sampling, in which there is a step named sampling from conditional distributions. I don't understand: 1. where is the conditional distribution from? From a general case, how can I get the conditional distribution?

Any help will be greatly appreciated.

  • $\begingroup$ You get the conditional distribution from your model. $\endgroup$ – Glen_b Aug 6 '13 at 10:01
  • $\begingroup$ Before trying to understand Gibbs sampling, you should learn the basics of probability theory and read up on conditional probability on your own. $\endgroup$ – Lucas Aug 6 '13 at 11:55

The goal is to generate samples for a multivariate distribution and its marginals, e.g. for the random vector $\mathbf{X} = (X_1, X_2, X_3)$, with pdf $f(x_1,x_2,x_3)$.

where is the conditional distribution from?

The conditional distributions, a.k.a. the full conditional distributions, are the distributions of each component random variable in the vector, conditional on the others, e.g. $X_1 | X_2=x_2, X_3=x_3$, and $X_2 | X_1=x_1, X_3=x_3$, and $X_3 | X_1=x_1, X_2=x_2$.

From a general case, how can I get the conditional distribution?

To get a full conditional, you treat the conditioning variables as constant in the multivariate pdf. E.g., to get the pdf for $X_1 | X_2=x_2, X_3=x_3$, you treat $x_2$ and $x_3$ as constant: the pdf $f(x_1|x_2,x_3)$ is obtained by simply starting with $f(x_1,x_2,x_3)$ and treating $x_2$ and $x_3$ as constant. What you hope is that you can recognize the core of the resulting pdf as a well-known univariate distribution, which you can easily sample from. Then, given a set of starting values, you sample from each full conditioning distribution in turn, using the most recent values of the conditioning variables.

  • $\begingroup$ You example is very helpful. I had another problem in learning Gibbs Sampling. It is hard for me to implement it. Could you suggest me some introductory books or notes that provide simple examples or codes to implement Latent Dirichlet allocation using Gibbs sampling? $\endgroup$ – user28859 Aug 6 '13 at 19:28
  • $\begingroup$ happy to have helped. Sorry I can't help with your follow-up question. Why not post another question on cross validated (giving a few more details of what you've tried / looked at so far)? $\endgroup$ – TooTone Aug 7 '13 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.