2
$\begingroup$

I understand the theory of Gibbs sampling. It is an iterative sampling algorithm that defines, sequence of random variables with the property of a Markov chain. Specifically, I choose any starting value, $(x_0, y_0)$ and discard the initial samples if my guess is poor (known as burn-in), then sample $x_1 \sim p(x|y_0)$ and $y_1 \sim p(y|x_1)$ for the first iteration. How do I derive the Gibbs sampler for the bivariate distribution with pdf:

$p(x,y) \propto \mathbb{1}(|x - y| < c) \mathbb{1}(x,y \in (0,1))$

In the examples I've seen online, like on page 3 of Casella's 1992 paper (http://biostat.jhsph.edu/~mmccall/articles/casella_1992.pdf), the conditional distributions can be written as a known distribution with varying parameters. In the bivariate pdf above, I don't see how the indicator functions can be interpreted as a known distribution.

Specifically, how do I solve for $p(x|y)$ and $p(y|x)$?

My work thus far:

\begin{equation} \begin{array}{lcl} p(x,y) &\propto & p(x|y) \\[2ex] &\propto & \mathbb{1}(-c < |x-y| < c) \mathbb{1}(x,y \in (0,1)) \\[2ex] &\propto & \mathbb{1}(-c +y < x < c + y) \mathbb{1}(x \in (0,1)) \\[2ex] p(x,y) &\propto & p(y|x) \\[2ex] &\propto & \mathbb{1}(-c < |x-y| < c) \mathbb{1}(x,y \in (0,1)) \\[2ex] &\propto & \mathbb{1}(-c - x < -y < c - x) \mathbb{1}(x,y \in (0,1)) \\[2ex] &\propto & \mathbb{1}(c + x < y < -c + x) \mathbb{1}(y \in (0,1)) \\[2ex] \end{array} \end{equation}

$\endgroup$
  • 1
    $\begingroup$ This seems to be routine bookwork as might be set for a class exercise. Please add the self-study tag and read its tag wiki. Then specify what you already understand and what guidance you need. $\endgroup$ – Glen_b -Reinstate Monica Mar 3 '15 at 8:02
  • $\begingroup$ The Q&A format is not a suitable way to teach you Gibbs sampling; we need specific questions. Do you understand what Gibbs sampling consists of? $\endgroup$ – Glen_b -Reinstate Monica Mar 3 '15 at 8:03
  • 1
    $\begingroup$ Is the problem actually understanding the notation $1(|x,y| < c)$ rather than Gibbs sampling? $\endgroup$ – Juho Kokkala Mar 3 '15 at 8:05
  • 1
    $\begingroup$ @Glen_b Corrected my question. I am trying to solve for the conditional distribution $p(x|y)$ but am having trouble translating the indicator function into a known distribution. $\endgroup$ – user2205916 Mar 3 '15 at 12:57
  • $\begingroup$ Note that Gibbs sampling is not about sequences of pairs; it works with no matter how many variables. $\endgroup$ – alberto Mar 3 '15 at 13:07
2
$\begingroup$

First, it is useful to understand what $p(x,y)$ is. The superposition of an area which is $1\times\text{constant}$ (the proportionality constant that makes everything integrate to 1) in every point where $x$ and $y$ are at distance $d<c$ (if you go through the $x$ axis, you draw a band from $x+c$ to $x-c$) and a square from $(0,0)$ to $(1,1)$.

If your imagination fails, there is Wolfram Alpha :)

joint_density

To get $p(x | y)$ you must get the slice of the figure taken at the given $y$. You see it is not the same slice at every $y$?. That means that, of course, $p(x | y)$ should include $y$ somewhere. Otherwise you are saying that $x$ is independent of $y$, that is $p(x | y) = p(x)$, and $p(x,y) = p(x)p(y)$, which would be a super boring joint distribution and you wouldn't need Gibbs sampling anymore.

Note: on "translating" the indicator function to a known distribution. What Gibbs sampling does is to attack every dimension separately. You don't know how to deal with the ugly bi-dimensional distribution of the figure. But you know how to deal with its uni-dimensional slices.

You just have to take the red pill and stop getting frightened by the bi-dimensional figure. We are doing Gibbs sampling, we live in 1-dimensional slices! ;)

$\endgroup$
  • $\begingroup$ Thanks. This was helpful for my understanding, Alberto. However, I'm wondering if my derived mathematical statements for the conditionals are acceptable answers. Examples I've seen have always put the conditionals in terms of a known distribution. $\endgroup$ – user2205916 Mar 3 '15 at 17:05
  • 1
    $\begingroup$ Your conditionals are wrong. The conditional on $y$ is a slice $[a,b]$ where $a$ and $b$ are defined with respect to $y$ (as the figure shows). Forget about probabilities and think geometrically. Then you'll see that the slice $[a,b]$ has the shape of a very, very, very trivial distribution. $\endgroup$ – alberto Mar 3 '15 at 17:18
  • 1
    $\begingroup$ Ask yourself the following question: imagine $p(x,y)$ is the Pentagon building (though it would rather be the Hexagon). It has a flat roof. If you cut it with giant plane through some $y$, what would be the shape of that cut? This would be your distribution (up to a constant factor, of course, you just have to make it integrate to 1). $\endgroup$ – alberto Mar 3 '15 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.