I want to learn Gibbs sampling for a Bayesian model. How can I sample the variable from the conditional distribution?
In this example, arrow means dependent; for example, Grade depends on Difficulty and Intelligence.
To use Gibbs sampling to calculate the joint distribution, first I set the Difficulty and Intelligence to (1,1).
The next step is to sample Grade from the $\rm{P(Grade|Difficulty=1,Intelligence=1)}$, but how can I sample?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ This depends on what the conditional distributions are, which in turn is determined by your model. More details needed. $\endgroup$– Juho KokkalaApr 2, 2014 at 14:49
-
1$\begingroup$ Could you edit the clarifying information into your question? $\endgroup$– Glen_bApr 2, 2014 at 21:23
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
Since we are calculating the joint distribution, we'll assume that our initial sample is $ x = P(D=0,I=0,G=0,L=0,S=0) $ .
To calculate the next sample, we'll need to sample each variable from the conditional distribution.
- $ P(D\mid G,I,S,L) $,from the conditional independencies in the Bayes net, simplifies to just sampling $ P(D)$. We sample and get the value $D=1$.
- Similarly for $ I $, we sample and get the value $ I=1$.
- Sampling for $ P(G\mid D,I,S,L) $, due to the conditional independencies encoded by the Bayes net, simplifies to $ P(G\mid D,I) $. Since we have already sampled $ D=1,I=1 $, we use those values and sample $ P(G\mid D=1,I=1) $. In the CPD for Grade, we can choose one of the value from the last row (where $ D=1,I=1 $). We sample and get the value $ G=2 $ (the value 0.3)
- $ P(L\mid I, G,D,S) $ simplifies to $ P(L\mid G) $. We sample from the second row the Letter CPD, where $ G=2 $, and we sample and get $ L=1 $ (the value 0.6).
- Similarly, sample $ P(S \mid I,L, G,D,) $ by simplifying to $ P(S \mid I) $. We get $ S=1 $ (sampling from the second row of the CPD where $ I=1 $.
And we'll have a new sample $ x': P(D=1,I=1,G=2,L=1,S=1) $.
-
$\begingroup$ Thank you very much for your answer! At the beginning I am not clear about how can I sample? I think it is not right to chose the one that with bigger P. The key point is to chose a properway to define its distribution according the given CPD. $\endgroup$ Apr 29, 2014 at 19:55
-
$\begingroup$ I didn't get what you meant by "it is not right to choose the one with the bigger P". In a CPD (such as $ P(L \mid G) $ ), any one of the assignments is chosen at random, not the one with the highest probability. $\endgroup$– shark8meApr 30, 2014 at 10:03
-
$\begingroup$ Yes, you are right, the assignments is chosen randomly based on the CPD :)) $\endgroup$ May 1, 2014 at 19:21
-
$\begingroup$ @shark8me When you say sample $P(D)$, to clarify, you mean sample a number from [0, 1] with uniform distribution, and use that number (probability) to map to a value of D using the associated CDF, right? $\endgroup$ May 13, 2016 at 20:48
$\begingroup$
$\endgroup$
If the conditional distribution cannot be generated directly from standard random generators, you can apply a Metroplolis-Hastings schema within the Gibbs sampler.
- draw one sample, $D_t$, from $P(D|G_{t-1},I_{t-1},S_{t-1},L_{t-1})$ using Metropolis sampler with initial value $D_{t-1}$
- draw one sample, $G_t$, from $P(G|D_{t},I_{t-1},S_{t-1},L_{t-1})$ using Metropolis sampler with initial value $G_{t-1}$
- draw one sample, $I_t$, from $P(I|D_{t},G_{t},S_{t-1},L_{t-1})$ using Metropolis sampler with initial value $I_{t-1}$
...
reference: Introduction Monte Carlo Methods with R chapter 7
