# Gibbs Sampling Inserting Some Known Predictors

Imagine you would like to use a simple Gibbs sampling to resample from a joint probability distribution which is difficult to model (but you know all the conditionals $Pr\left(X_i|X_1,...,X_{i-1},X_{i+1},...,X_N\right)$, for $i=1,...,N$, hence the Gibbs sampling). At the end, you would have a set of many drawn vectors of variables $\left(x_1^{(k)},x_2^{(k)},...,x_N^{(k)}\right)$, for $k=1,...,K$.

Imagine now you would like to assign a vector of variables $\left(y_1^{(k)},y_2^{(k)},...,y_M^{(k)}\right)$ to each of the already drawn vectors. You have some conditional probabilities like $Pr\left(Y_j|Y_1,...,Y_{j-1},Y_{j+1},...,Y_M,X_1,...,X_N\right)$. Again, modelling joint distributions is difficult.

Do you think it can work a Gibbs sampling with these conditionals by inserting sequentially the previously drawn $x_i^{(k)}$ and, after discarding some steps, collecting the sampled $y_j^{(k)}$? Otherwise, do you have a strategy to apply in these cases?

Thanks!

It really does not. The first set of draw are from the joint distribution of $(X_1, \dots , X_N)$. Now if you sample from $(Y_j | Y_1, \dots , Y_M, X_1, \dots , X_n)$ in fact your sample is from the distribution of $(Y_j | Y_1, \dots , Y_M)$. What you are really doing is compute the integral: $$f(Y_1| Y_1,\dots, Y_M) = \int f(Y_1| Y_1,\dots, Y_M X_1, \dots ,X_n)f(X_1, \dots , X_n) dX_1, \dots D X_n$$ It is well know that if you sample from $f(X_1, \dots , X_n)$ (your first set of samples) and then you sample from $f(Y_1| Y_1,\dots, Y_M X_1, \dots ,X_n)$ it is a marginalization and then you have a sample from $f(Y_1| Y_1,\dots, Y_M)$
• Thank you very much for your answer. What do you suggest to do in these cases, when you have to assign some new vectors $\left(y_1^{(k)},y_2^{(k)},...,y_M^{(k)}\right)$ to some already generated $\left(x_1^{(k)},x_2^{(k)},...,x_N^{(k)}\right)$? Would you use simple conditionals like $Pr\left(Y_j|X_1,...,X_N\right)$? – Pippo Jul 12 '14 at 8:48
• What distribution do you need: $f(Y_1, \dots , Y_m)$, $f(Y_1, \dots , Y_m, X_1, \dots , X_n)$, $f(Y_1, \dots , Y_m |X_1, \dots ,X_n)$? – niandra82 Jul 12 '14 at 10:49
• The second one. Would it be possible to generate vectors $\left(y_1^{(k)},y_2^{(k)},...,y_M^{(k)}\right)$, each one assigned to a $\left(x_1^{(k)},x_2^{(k)},...,x_N^{(k)}\right)$, given some conditionals and without directly accessing $f\left(Y_1,...,Y_m,X_1,...,X_n\right)$? – Pippo Jul 12 '14 at 19:50
• I think it is impossible. When you simulate $X_1, \dots X_n$ you have no information about $Y_1, \dots Y_m$ so it is unlikely that the draw $x_i^{(k)}$ can be considered as a subset of a sample from $f(Y_1, \dots Y_m, X_1, \dots X_n)$. – niandra82 Jul 12 '14 at 21:32