Imagine you would like to use a simple Gibbs sampling to resample from a joint probability distribution which is difficult to model (but you know all the conditionals $Pr\left(X_i|X_1,...,X_{i-1},X_{i+1},...,X_N\right)$, for $i=1,...,N$, hence the Gibbs sampling). At the end, you would have a set of many drawn vectors of variables $\left(x_1^{(k)},x_2^{(k)},...,x_N^{(k)}\right)$, for $k=1,...,K$.

Imagine now you would like to assign a vector of variables $\left(y_1^{(k)},y_2^{(k)},...,y_M^{(k)}\right)$ to each of the already drawn vectors. You have some conditional probabilities like $Pr\left(Y_j|Y_1,...,Y_{j-1},Y_{j+1},...,Y_M,X_1,...,X_N\right)$. Again, modelling joint distributions is difficult.

Do you think it can work a Gibbs sampling with these conditionals by inserting sequentially the previously drawn $x_i^{(k)}$ and, after discarding some steps, collecting the sampled $y_j^{(k)}$? Otherwise, do you have a strategy to apply in these cases?



It really does not. The first set of draw are from the joint distribution of $(X_1, \dots , X_N)$. Now if you sample from $(Y_j | Y_1, \dots , Y_M, X_1, \dots , X_n)$ in fact your sample is from the distribution of $(Y_j | Y_1, \dots , Y_M)$. What you are really doing is compute the integral: $$ f(Y_1| Y_1,\dots, Y_M) = \int f(Y_1| Y_1,\dots, Y_M X_1, \dots ,X_n)f(X_1, \dots , X_n) dX_1, \dots D X_n $$ It is well know that if you sample from $f(X_1, \dots , X_n)$ (your first set of samples) and then you sample from $f(Y_1| Y_1,\dots, Y_M X_1, \dots ,X_n)$ it is a marginalization and then you have a sample from $f(Y_1| Y_1,\dots, Y_M) $

  • $\begingroup$ Thank you very much for your answer. What do you suggest to do in these cases, when you have to assign some new vectors $\left(y_1^{(k)},y_2^{(k)},...,y_M^{(k)}\right)$ to some already generated $\left(x_1^{(k)},x_2^{(k)},...,x_N^{(k)}\right)$? Would you use simple conditionals like $Pr\left(Y_j|X_1,...,X_N\right)$? $\endgroup$ – Pippo Jul 12 '14 at 8:48
  • $\begingroup$ What distribution do you need: $f(Y_1, \dots , Y_m)$, $f(Y_1, \dots , Y_m, X_1, \dots , X_n)$, $f(Y_1, \dots , Y_m |X_1, \dots ,X_n)$? $\endgroup$ – niandra82 Jul 12 '14 at 10:49
  • $\begingroup$ The second one. Would it be possible to generate vectors $\left(y_1^{(k)},y_2^{(k)},...,y_M^{(k)}\right)$, each one assigned to a $\left(x_1^{(k)},x_2^{(k)},...,x_N^{(k)}\right)$, given some conditionals and without directly accessing $f\left(Y_1,...,Y_m,X_1,...,X_n\right)$? $\endgroup$ – Pippo Jul 12 '14 at 19:50
  • $\begingroup$ I think it is impossible. When you simulate $X_1, \dots X_n$ you have no information about $Y_1, \dots Y_m$ so it is unlikely that the draw $x_i^{(k)}$ can be considered as a subset of a sample from $f(Y_1, \dots Y_m, X_1, \dots X_n)$. $\endgroup$ – niandra82 Jul 12 '14 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.