I want to learn Gibbs sampling for a Bayesian model. How can I sample the variable from the conditional distribution?
a Bayesian model
In this example, arrow means dependent; for example, Grade depends on Difficulty and Intelligence. To use Gibbs sampling to calculate the joint distribution, first I set the Difficulty and Intelligence to (1,1).
The next step is to sample Grade from the $\rm{P(Grade|Difficulty=1,Intelligence=1)}$, but how can I sample?

  • 1
    $\begingroup$ This depends on what the conditional distributions are, which in turn is determined by your model. More details needed. $\endgroup$ – Juho Kokkala Apr 2 '14 at 14:49
  • $\begingroup$ Could you edit the clarifying information into your question? $\endgroup$ – Glen_b Apr 2 '14 at 21:23

Since we are calculating the joint distribution, we'll assume that our initial sample is $ x = P(D=0,I=0,G=0,L=0,S=0) $ .

To calculate the next sample, we'll need to sample each variable from the conditional distribution.

  1. $ P(D\mid G,I,S,L) $,from the conditional independencies in the Bayes net, simplifies to just sampling $ P(D)$. We sample and get the value $D=1$.
  2. Similarly for $ I $, we sample and get the value $ I=1$.
  3. Sampling for $ P(G\mid D,I,S,L) $, due to the conditional independencies encoded by the Bayes net, simplifies to $ P(G\mid D,I) $. Since we have already sampled $ D=1,I=1 $, we use those values and sample $ P(G\mid D=1,I=1) $. In the CPD for Grade, we can choose one of the value from the last row (where $ D=1,I=1 $). We sample and get the value $ G=2 $ (the value 0.3)
  4. $ P(L\mid I, G,D,S) $ simplifies to $ P(L\mid G) $. We sample from the second row the Letter CPD, where $ G=2 $, and we sample and get $ L=1 $ (the value 0.6).
  5. Similarly, sample $ P(S \mid I,L, G,D,) $ by simplifying to $ P(S \mid I) $. We get $ S=1 $ (sampling from the second row of the CPD where $ I=1 $.

And we'll have a new sample $ x': P(D=1,I=1,G=2,L=1,S=1) $.

  • $\begingroup$ Thank you very much for your answer! At the beginning I am not clear about how can I sample? I think it is not right to chose the one that with bigger P. The key point is to chose a properway to define its distribution according the given CPD. $\endgroup$ – user2991640 Apr 29 '14 at 19:55
  • $\begingroup$ I didn't get what you meant by "it is not right to choose the one with the bigger P". In a CPD (such as $ P(L \mid G) $ ), any one of the assignments is chosen at random, not the one with the highest probability. $\endgroup$ – shark8me Apr 30 '14 at 10:03
  • $\begingroup$ Yes, you are right, the assignments is chosen randomly based on the CPD :)) $\endgroup$ – user2991640 May 1 '14 at 19:21
  • $\begingroup$ @shark8me When you say sample $P(D)$, to clarify, you mean sample a number from [0, 1] with uniform distribution, and use that number (probability) to map to a value of D using the associated CDF, right? $\endgroup$ – Jane Wayne May 13 '16 at 20:48

If the conditional distribution cannot be generated directly from standard random generators, you can apply a Metroplolis-Hastings schema within the Gibbs sampler.

  1. draw one sample, $D_t$, from $P(D|G_{t-1},I_{t-1},S_{t-1},L_{t-1})$ using Metropolis sampler with initial value $D_{t-1}$
  2. draw one sample, $G_t$, from $P(G|D_{t},I_{t-1},S_{t-1},L_{t-1})$ using Metropolis sampler with initial value $G_{t-1}$
  3. draw one sample, $I_t$, from $P(I|D_{t},G_{t},S_{t-1},L_{t-1})$ using Metropolis sampler with initial value $I_{t-1}$


reference: Introduction Monte Carlo Methods with R chapter 7


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.