# Gibbs sampling and mixed distribution

For a project, I need to simulate from a joint distribution with both continuous and discrete variables that are dependent. The conditional distribution of any variable given the rest is known. I decided to simulate with the help of Gibbs sampling. My question is:

Is it possible to simulate from a joint distribution containing both continuous and discrete variable with Gibbs sampling?

The example by Glen_b makes the point that yes you can indeed implement a Gibbs sampler where one component is discrete and the other is continuous. I present an example where you need to resort to Gibbs sampling to draw from the posterior.

Consider the Bayesian variable selection model of George and McCulloch. They present the following Bayesian model \begin{align*} Y|X,\beta, \sigma^2 &\sim N(X\beta, \sigma^2I)\\ \beta_i|\gamma_i = 1 &\sim N(0, c_i^2\tau_i^2)\\ \beta_i| \gamma_i = 0 &\sim N(0, \tau^2_l)\\ \sigma^2 &\sim \text{Inverse Gamma}(v/2, v \lambda/2)\\ P(\gamma_i = 1) &= 1 - P(\gamma_i = 0) = p_i. \end{align*}

Here, $\tau_i^2, c_i^2, \tau_l^2 v, \lambda$ are fixed. The parameters to be estimated are $\gamma$ which are discrete and $\beta$ and $\sigma^2$. The posterior distribution is intractable, so we resort to MCMC techniques such as Gibbs sampling. As you mentioned, for that we need the full conditionals. The paper provides the following conditionals. Let $A = \left(\sigma^{-2}X^TX + D^{-2} \right)^{-1}$, where $D$ is a diagonal matrix dependent on $\gamma$. \begin{align*} \beta|\gamma, \sigma^2 & \sim N_p(\sigma^{-2}AX^Ty, A)\\ \sigma^2 |\beta, \gamma & \sim \text{Inverse Gamma}\left(\dfrac{n+v}{2}, \dfrac{(Y - X\beta)^T(y - X\beta) + v\lambda}{2} \right)\\ P(\gamma_i = 1|\beta, \sigma^2) & = \dfrac{p_i f(\beta|\sigma^2, \gamma_i = 1)f(\sigma^2|\gamma_i = 1)}{p_i f(\beta|\sigma^2, \gamma_i = 1)f(\sigma^2|\gamma_i = 1) + (1-p_i)f(\beta|\sigma^2, \gamma_i = 0)f(\sigma^2|\gamma_i = 0)} \end{align*}

Notice that the conditional of $\gamma$ looks complicated but is not too bad if you plug in the distributions. One run through this Gibbs sampler, updates $\beta$ from a normal, $\sigma^2$ from an Inverse Gamma and $\gamma$ by tossing a coin with the probability indicated.

• In the example you provided, the variables follow a certain order, X depends on Y but Y doesn't depend on X. What if they depend on one another? – Anthony Hauser May 4 '16 at 7:39
• @Anthony (Note for others reading along -- here the continuous variables are the $\beta$s and $\sigma$, and the discrete ones are the $\gamma$s.) It's quite convenient (natural) to *write* the model with the distribution of the $\beta_i$'s conditioned on the values of $\gamma_i$s and $\sigma$, but the distribution of $\gamma$ also depends on $\beta$ and $\sigma$ and $\sigma$ also depends on $\beta$ (well it's not independent of $\gamma$ but once you know $\beta$ that captures all of it) . These dependences are explicit in the equations above. – Glen_b May 4 '16 at 8:59

Yes, this presents no difficulty.

As long as you can sample from the full conditionals (and it sounds like you can) then yes.

For a bivariate $(U,V)$ that's just sampling $(V|U=u)$ and $(U|V=v)$. Let's consider a simple case (for which we don't really need Gibbs sampling). Let:

$f_{X,Q}(x,q)= \frac{{n\choose x}} {\mathrm{B}(\alpha,\beta)} q^{x+\alpha-1} (1-q)^{n-x+\beta-1}\,, \quad x=0,1,...,n \quad 0<q<1$

which can be written as

$f_{X,Q}(x,q) = f_{X|Q=q}(x) f_Q(q)$

$\qquad\qquad\:\:= {n\choose x}q^x(1-q)^{n-x}\,\cdot\,\frac{1} {\mathrm{B}(\alpha,\beta)} q^{\alpha-1} (1-q)^{\beta-1}$

So if we know $Q=q$ we can sample from $X$; it's just binomial.

On the other hand, conditional on $X=x$ we can see that $Q$ just has a beta distribution, so again we can sample from that.

[If we perform Gibbs sampling on that pair of full conditionals we'd be ultimately sampling from a beta-binomial marginal distribution for $X$; if that marginal was of primary interest we could calculate it directly by integration.]

• In the example you provided, the variables follow a certain order, X depends on Y but Y doesn't depend on X. What if they depend on one another? – Anthony Hauser May 4 '16 at 7:40
• @AnthonyHauser you need to clarify that remark - (in)dependence of random variables is symmetric - X and Y are independent if and only if Y and X are independent - so it is unclear what you mean by "Y doesn't depend on X". – Juho Kokkala May 4 '16 at 7:51
• @Anthony I'm not sure what you're getting at. The conditional distribution for Q does depend on X. The joint just happens to be a choice for which the conditional for $X|Q$ is of a particularly convenient form. This is quite common with models we use Gibbs sampling on -- typically for some of the variables it's quite natural to write the joint in a conditional times marginal form. This in no way suggests that there's some deficiency in the example. – Glen_b May 4 '16 at 8:44
• @JuhoKokkala Yes, my remark wasn't clear. What I mean is that the conditional distribution X|Y depends on Y and Y|X depends on X. In the example of Glen_b, it is not the case. – Anthony Hauser May 4 '16 at 8:53
• @anthony In my example the conditional for $Q$ certainly depends on $X=x$ (it's $\text{Beta}(x+\alpha,n-x+\beta)$. If you want to ask a new question about a specific example, it might be best in a new post at this point. – Glen_b May 4 '16 at 9:08