# Deriving Gibbs sampler for specific mixture model

Let $\theta_i$ be an indicator which is $0$ if score, $X_i$, is the same for both opponents, $1$ if different:

• $X_i|\theta_i \stackrel{\text{ind}}{\sim} (1-\theta_i) U(0, 1) + \theta_i Beta(1, \beta)$
• $\theta_i \stackrel{\text{iid}}{\sim}\text{Bernoulli}(\pi)$
• $\pi \sim \text{Uniform}(0,1)$
• $\beta \sim \text{Gamma}(\text{shape} = 0.001, \text{rate} = 0.001)$

Attempt: We need to find the following conditional posteriors which I am having trouble with. I have came across combining normal distributions which I understand, what is the approach for these more complicated distributions (any references would also be appreciated)?

$$\theta_i|X_i \sim$$ $$\pi|X_i \sim$$ $$\beta|X_i \sim$$

• The Gibbs sampler for mixtures is one of the first examples of applying the Gibbs sampler and a straightforward one. Can you tell us where you get stuck in deriving the joint distribution of $\pi,\beta,\theta_1,\ldots,\theta_n$? – Xi'an Dec 13 '15 at 11:20

Here is how we explain the resolution in Bayesian Essentials with R rephrased for your problem (but you can find the same approach in most textbooks):

With each $x_i$ is associated a missing variable $z_i$ [your $\theta_i$] that indicates its component. Formally, this means that we have a hierarchical structure associated with the model: $$z_i|\pi\sim\mathscr{B}(\pi)$$ and $$x_i|z_i=1,\beta \sim \beta x^{\beta-1}\qquad x_i|z_i=0,\beta \sim \mathcal{U}(01,1)$$ The completed likelihood corresponding to the missing structure is such that $$\ell(\beta,\pi|\mathbf{x},\mathbf{z})=\prod_{i=1}^n \pi^{z_i}(1-\pi)^{1-z_i}\,(1-x_i)^{(\beta-1) z_i}\beta^{z_i}$$ and the corresponding posterior is $$\pi(\beta,\pi|\mathbf{x},\mathbf{z}) \propto \beta^{-.999}\exp(-0.001\beta)\,\prod_{i=1}^n \pi^{z_i}(1-\pi)^{1-z_i}\,(1-x_i)^{(\beta-1) z_i}\beta^{z_i},$$ where $\mathbf{z}=(z_1,\ldots,z_n)$.

As I dislike very much the BUGS type priors with almost zero parameters, I will now use $\beta^{-1}$ instead of $\beta^{-.999}\exp(-0.001\beta)$

Using this completion, the joint posterior distribution of $(\beta,\pi)$ can be written in closed form as $$f(\beta,\pi|\mathbf{x},\mathbf{z}) \propto \pi^{\sum_{i=1}^nz_i}(1-\pi)^{n-\sum_{i=1}^nz_i}\,\beta^{\sum_{i=1}^n z_i-1}\left[\prod\nolimits_{z_i=1}(1-x_i)\right]^{\beta-1},$$ If $\pi$ and $\beta$ are independent a priori, then, given $\mathbf{z}$, the vectors $\pi$ and $\mathbf{x}$ are independent; that is, $f(\pi|\mathbf{x},\mathbf{z})=f(\pi|\mathbf{z})$. Moreover, in that case, $\beta$ is also independent a posteriori from $\pi$ given $\mathbf{x}$ and $\mathbf{z}$, with density $f(\beta|\mathbf{x},\mathbf{z})$. If we apply the Gibbs sampler in this problem, it involves the successive simulation of $\mathbf{z}$ and $(\pi,\beta)$ conditional on one another and on the data:

Initialization: Choose $\pi^{(0)}$ and $\beta^{(0)}$ arbitrarily.

Iteration $t$ $(t\ge 1)$:

1. For $i=1,\ldots,n$, generate $z_i^{(t)}$ such that $$\mathbb{P}\left(z_i=1\right)\propto \pi^{(t-1)}f\left(x_i|\beta^{(t-1)},z_i\right)$$
2. Generate $\pi^{(t)}$ according to $f(\pi|\mathbf{z}^{(t)})$.
3. Generate $\beta^{(t)}$ according to $f(\beta|\mathbf{z}^{(t)},\mathbf{x})$.

Try to solve the full conditionals before reading further

In your specific case, you can derive the full conditional $f(\pi|\mathbf{z})$ and $f(\beta|\mathbf{z},\mathbf{x})$ from the joint above: \eqalign{ f(\beta,\pi|\mathbf{x},\mathbf{z}) &\propto \pi^{\sum_{i=1}^nz_i}(1-\pi)^{n-\sum_{i=1}^nz_i}\,\beta^{\sum_{i=1}^n z_i-1}\left[\prod\nolimits_{z_i=1}(1-x_i)\right]^{\beta-1}\\ &\propto f(\pi|\mathbf{z}) f(\beta|\mathbf{x},\mathbf{z})\\ }

If you separate the terms in $\pi$ and the terms in $\beta$ you get $$f(\pi|\mathbf{z})\propto\pi^{\sum_{i=1}^n z_i}(1-\pi)^{n-\sum_{i=1}^nz_i}$$ which corresponds to a Beta $$\mathcal{B}e\left(1+\sum\nolimits_{i=1}^n z_i,1+n-\sum\nolimits_{i=1}^n z_i\right)$$ distribution and $$f(\beta|\mathbf{x},\mathbf{z})\propto\beta^{\sum_{i=1}^n z_i-1}\left[\prod\nolimits_{z_i=1}(1-x_i)\right]^{\beta}= \beta^{\sum_{i=1}^n z_i-1} \exp\left[-\beta\sum\nolimits_{z_i=1} \log(1-x_i)\right]$$ which corresponds to a Gamma $$\mathcal{Ga}\left(\sum\nolimits_{i=1}^n z_i,\sum\nolimits_{z_i=1} \log(1-x_i)\right)$$ distribution.

• Thank you for the detailed explanation, I am working through the details. – user98056 Dec 14 '15 at 5:30