I want to learn about how to do Gibbs sampling, starting with finding conditional distributions given a joint distribution. While looking for examples, I found this blog post that I wanted to replicate on my own, but I am having trouble understanding how to algebraically find the conditional distributions given the joint distribution.

The joint distribution is $$ f(x, y) = kx^2 exp(-xy^2 - y^2 + 2y - 4x) $$ where $k$ is a normalization constant. The marginal distributions are $$x \mid y \sim Gamma(3, y^2 + 4) $$ and $$y \mid x \sim N(\frac{1}{x + 1}, \frac{1}{2(x + 1)}) $$

What are the steps to derive the two marginals from the joint? Is there a trick I missing? I feel like since it was unexplained in the example it should be somewhat simple but I am just not seeing it...

Thanks! Also, bonus points if anyone can point me to any (free to access) examples with solutions similar to this. I want to practice deriving conditional distributions from joint ones.


Those distributions you call "marginal" are not marginal. They are conditional distributions because you wrote $x \mid y$. The marginal distribution of $X$, for example, is necessarily independent of the value of $Y$.

To see how the conditional distribution is gamma, all you have to do is write $$f_{X \mid Y}(x) = \frac{f_{X,Y}(x,y)}{f_Y(y)} \propto f_{X,Y}(x,y).$$ That is to say, the conditional distribution is proportional to the joint distribution, appropriately normalized. So we have $$f_{X \mid Y}(x) \propto x^2 e^{-x(y^2+4)},$$ completely ignoring any factors that are not functions of $x$. Then we recognize that the gamma distribution has density $$f_S(s) \propto s^{a-1} e^{-bs},$$ so the choice of shape $a = 3$ and rate $b = y^2+4$ demonstrates that the conditional distribution $X \mid Y \sim \operatorname{Gamma}(a = 3, b = y^2+4)$.

The conditional distribution of $Y \mid X$ is done in a similar fashion. Just ignore constants of proportionality: $$f_{Y \mid X}(y) \propto e^{-(x+1)y^2+2y},$$ but this one requires us to complete the square to get it to look like a normal density: $$-(x+1)y^2 + 2y = (x+1)\left(-\left(y - \tfrac{1}{x+1}\right)^2 \right) + \tfrac{1}{x+1},$$ and after exponentiating and removing the $e^{1/(x+1)}$ factor, comparing this against $$f_W(w) \propto e^{-(w-\mu)^2/(2\sigma^2)},$$ we see that we have a normal density with mean $\mu = 1/(x+1)$ and variance $\sigma^2 = 1/(2(x+1))$.

Now, if you wanted the marginal distributions, you would need to integrate: $$f_X(x) = \int_{y=-\infty}^\infty f_{X,Y}(x,y) \, dy,$$ for example. And as you can see, this expression will not be a function of $Y$. The difference is that if I simulated realizations of ordered pairs $(X_i, Y_i)$ from the joint distribution, the marginal density for $X$ would be what you would see if I only told you the values of $X_i$. The conditional distribution of $X$ given $Y = y$ would be what you would see if I only gave you the $X_i$ for which the corresponding $Y_i$ was equal to $y$.


The "trick" is to observe that $f(x\mid y)=f(x,y)/f(y)$ is proportional to $f(x,y)$ up to terms that do not involve $x$. Hence, $f(x\mid y)\propto x^2\exp(-(y^2+4)x)$, and this is the "kernel" of a $\mathrm{Gamma}(3,y^2+4)$ density. The other full conditional $f(y\mid x)$ is obtained similarly after completing the square in the exponent.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.