I want to learn Gibbs sampling for a Bayesian model. How can I sample the variable from the conditional distribution?
a Bayesian model
In this example, arrow means dependent; for example, Grade depends on Difficulty and Intelligence. To use Gibbs sampling to calculate the joint distribution, first I set the Difficulty and Intelligence to (1,1).
The next step is to sample Grade from the $\rm{P(Grade|Difficulty=1,Intelligence=1)}$, but how can I sample?

  • 1
    $\begingroup$ This depends on what the conditional distributions are, which in turn is determined by your model. More details needed. $\endgroup$ Commented Apr 2, 2014 at 14:49
  • 1
    $\begingroup$ Could you edit the clarifying information into your question? $\endgroup$
    – Glen_b
    Commented Apr 2, 2014 at 21:23

2 Answers 2


Since we are calculating the joint distribution, we'll assume that our initial sample is $ x = P(D=0,I=0,G=0,L=0,S=0) $ .

To calculate the next sample, we'll need to sample each variable from the conditional distribution.

  1. $ P(D\mid G,I,S,L) $,from the conditional independencies in the Bayes net, simplifies to just sampling $ P(D)$. We sample and get the value $D=1$.
  2. Similarly for $ I $, we sample and get the value $ I=1$.
  3. Sampling for $ P(G\mid D,I,S,L) $, due to the conditional independencies encoded by the Bayes net, simplifies to $ P(G\mid D,I) $. Since we have already sampled $ D=1,I=1 $, we use those values and sample $ P(G\mid D=1,I=1) $. In the CPD for Grade, we can choose one of the value from the last row (where $ D=1,I=1 $). We sample and get the value $ G=2 $ (the value 0.3)
  4. $ P(L\mid I, G,D,S) $ simplifies to $ P(L\mid G) $. We sample from the second row the Letter CPD, where $ G=2 $, and we sample and get $ L=1 $ (the value 0.6).
  5. Similarly, sample $ P(S \mid I,L, G,D,) $ by simplifying to $ P(S \mid I) $. We get $ S=1 $ (sampling from the second row of the CPD where $ I=1 $.

And we'll have a new sample $ x': P(D=1,I=1,G=2,L=1,S=1) $.

  • $\begingroup$ Thank you very much for your answer! At the beginning I am not clear about how can I sample? I think it is not right to chose the one that with bigger P. The key point is to chose a properway to define its distribution according the given CPD. $\endgroup$ Commented Apr 29, 2014 at 19:55
  • $\begingroup$ I didn't get what you meant by "it is not right to choose the one with the bigger P". In a CPD (such as $ P(L \mid G) $ ), any one of the assignments is chosen at random, not the one with the highest probability. $\endgroup$
    – shark8me
    Commented Apr 30, 2014 at 10:03
  • $\begingroup$ Yes, you are right, the assignments is chosen randomly based on the CPD :)) $\endgroup$ Commented May 1, 2014 at 19:21
  • $\begingroup$ @shark8me When you say sample $P(D)$, to clarify, you mean sample a number from [0, 1] with uniform distribution, and use that number (probability) to map to a value of D using the associated CDF, right? $\endgroup$
    – Jane Wayne
    Commented May 13, 2016 at 20:48

If the conditional distribution cannot be generated directly from standard random generators, you can apply a Metroplolis-Hastings schema within the Gibbs sampler.

  1. draw one sample, $D_t$, from $P(D|G_{t-1},I_{t-1},S_{t-1},L_{t-1})$ using Metropolis sampler with initial value $D_{t-1}$
  2. draw one sample, $G_t$, from $P(G|D_{t},I_{t-1},S_{t-1},L_{t-1})$ using Metropolis sampler with initial value $G_{t-1}$
  3. draw one sample, $I_t$, from $P(I|D_{t},G_{t},S_{t-1},L_{t-1})$ using Metropolis sampler with initial value $I_{t-1}$


reference: Introduction Monte Carlo Methods with R chapter 7


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.