# Conditional Probability - Mixture Model

I know that the likelihood in a p-dimensional Gaussian mixture model is given by

$$p(s|\theta) = \sum_{b_1 = 0}^1\cdots\sum_{b_p = 0}^1\left[ \prod_{i=1}^pw^{1-b_i}(1-w)^{1-b_i}\right]\phi_p(s|\mu(b,\theta),\Sigma)$$

where $$\phi_p(x|a,B)$$ denota a p-dimensional Gaussian density function, $$w \in [0,1]$$ is a mixture weight, $$\mu(b,\theta)' = ((1-2b_1)\theta_1, \cdots, (1-2b_p)\theta_p)$$, $$b_i \in \{0,1\}$$ and $$\Sigma = [\Sigma_{i,j}]$$ is such that $$\Sigma_{i,i} = 1$$ and $$\Sigma_{i,j} = \rho$$ for $$i \neq j$$.

For $$p = 2$$, let $$s \sim N(\mu, \Sigma)$$ $$\mu_i = (1-2b_i)\theta_i$$ $$b_i \sim Bern(1-w)$$ $$\theta_i \sim U(-20, 40)$$

You can demonstrate that the full conditional distribution for $$\theta_1$$ and $$b_1$$, respectively, are given by

$$p(\theta_1|\theta_2,b,s) \sim N\left(\mu_{\theta_1}, \sqrt{1-\rho^2}\right)$$

where $$\mu_{\theta_1} = [\rho((1-2b_2)\theta_2 - s_2 + s_1)/(1-2b_1)]$$.

In the model above, replacing $$\theta_1$$ with $$u_1 + u_2$$ and $$\theta_2$$ with $$u_1 - u_2$$, where $$u_1, u_2 \sim U(-20, 40)$$ can you still guarantee normality?

I don't know if I'm misinterpreting, but when replacing a uniform with a (triangular) uniform sum, would the convergence to a normal distribution not have to be faster? I did some simulation studies and did not get a satisfactory result with this hypothesis. Thanks in advance!

$$\theta_1, \theta_2 \sim U(-20,40) \quad \theta_1\perp \theta_2$$

Under the assumption of independence between,

$$f_{\theta_1, \theta_2}(\theta_1, \theta_2) = \frac{1}{3600}\mathbb{I}_{\{-20\leq \theta_1 \leq 40, -20 \leq \theta_2 \leq 40\}}(\theta_1, \theta_2)$$

Making variable changes

$$\begin{cases} \alpha_1 = g_1(\theta_1, \theta_2) = \theta_1 + \theta_2\\ \alpha_2 = g_2(\theta_1, \theta_2) = \theta_1 - \theta_2 \end{cases} \Rightarrow \begin{cases} \theta_1 = \frac{\alpha_1 + \alpha_2}{2}\\ \theta_2 = \frac{\alpha_1 - \alpha_2}{2}\\ \end{cases} \qquad |J| = \frac{1}{2}$$

\begin{align*} f_(\alpha_1, \alpha_2) = & \:f_{\theta_1, \theta_2}(g_1^{-1}, g_2^{-1})|J|\\ = & \:\frac{1}{3600}\mathbb{I}_{\left\{-20\leq \frac{\alpha_1 + \alpha_2}{2} \leq 40, -20\leq \frac{\alpha_1 - \alpha_2}{2} \leq 40 \right\}}(\alpha_1, \alpha_2)\frac{1}{2} \end{align*}

The variation set follows as below.

$$A = \left\{(\alpha_1, \alpha_2);\:-20\leq \frac{\alpha_1 + \alpha_2}{2} \leq 40, -20\leq \frac{\alpha_1 - \alpha_2}{2} \leq 40 \right\}$$

The marginal of $$\alpha_1$$ is given by

\begin{align*} f_{\alpha_1}(\alpha_1) = & \: \int_{-\infty}^\infty f(\alpha_1, \alpha_2)\:d\alpha_2\\ = & \: \begin{cases} \frac{\alpha_1 + 40}{3600},\: -40\leq \alpha_1 \leq 20\\ \frac{80 - \alpha_1}{3600},\: 20\leq \alpha_1 \leq 80\\ \end{cases} \end{align*}

Now

\begin{align*} p(\alpha_1 | \alpha_2, b, s) \propto & \: p(s | \alpha, b, s) p(\alpha_1)\\ = & \: \begin{cases} \phi_2(s | \mu(b, \alpha),\Sigma) \frac{\alpha_1 + 40}{3600},\: -40 \leq \alpha_1 \leq 20\\ \phi_2(s | \mu(b, \alpha),\Sigma) \frac{80 - \alpha_1}{3600},\: 20 \leq \alpha_1 \leq 80 \end{cases} \end{align*}

That when solving $$p(\alpha_1 | \alpha_2, b, s)$$ in $$\alpha_1$$, note that the result will be as follows $$f(\alpha_1)e^{-g(\alpha_1)}$$ That runs away from the core of a normal.

Is there any misconception?

• You didn't specify the prior for $w$ or $\Sigma$ – Taylor Sep 14 '19 at 14:05
• I presume the inner likelihood term should be$$\prod_i\omega^{b_i}(1-\omega)^{1-b_i}$$ – Xi'an Sep 14 '19 at 14:39
• @Taylor, I believe the above model is already well specified. For example, $s' = (2,2), w = 0,3$ and $\rho = 0,3$. are values for these parameters. – Jackson Maike Sep 14 '19 at 15:23
• @Xi'an, the post from which I took the model points out normality when I wear a uniform for $\theta$. My idea is that if I use a $f(\theta)$ in which its distribution is symmetrical, such as triangular, normality would still have to be maintained. Am I right? Anyway I will try to demonstrate analytically. – Jackson Maike Sep 14 '19 at 15:27
• Given the peculiarity of the model (which is not a standard mixture model since the number of components grows exponentially in $p$), could you indicate a relevant reference or a motivation for this model? – Xi'an Sep 15 '19 at 14:21

This is a very peculiar kind of mixture distribution in that, for each component $$s_i$$ $$(i=1,\ldots,p)$$ of a random realisation $$\mathbf s=(s_1,\ldots,s_p)$$ from this distribution, the mean is either $$\theta_i$$ or $$-\theta_i$$ with probabilities $$\omega$$ and $$1-\omega$$, independently. Formally, this makes a mixture with $$2^p$$ components. If $$\boldsymbol \theta$$ is the only (unknown) parameter of the model the posterior distribution of $$\boldsymbol \theta$$ conditional on one single observation $$\mathbf s$$ and the associated allocation latent variable $$\mathbf b=(b_1,\ldots,b_p)\in\{-1,1\}^p$$ is the product of the prior on $$\boldsymbol \theta$$ by a Normal likelihood $$\varphi(\mathbf s|\mathbf b\cdot\boldsymbol \theta,\Sigma)$$ (where $$\mathbf b\cdot\boldsymbol \theta$$ denotes a component-wise product).
Were the prior to be flat, the posterior would then be a Normal distribution $$\varphi(\boldsymbol \theta|\mathbf b\cdot\mathbf s,\mathbf b^\prime\cdot\Sigma\cdot\mathbf b)$$ since \begin{align*}(\mathbf s-\mathbf b\cdot\boldsymbol \theta)^\prime\Sigma^{-1}(\mathbf s-\mathbf b\cdot\boldsymbol \theta)&=(\mathbf b\cdot\mathbf b\cdot\mathbf s-\mathbf b\cdot\boldsymbol \theta)^\prime\Sigma^{-1}(\mathbf b\cdot\mathbf b\cdot\mathbf s-\mathbf b\cdot\boldsymbol \theta)\\&=(\mathbf b\cdot\mathbf s-\boldsymbol \theta)^\prime\mathbf b^\prime\cdot\Sigma^{-1}\cdot\mathbf b(\mathbf b\cdot\mathbf s-\boldsymbol \theta)\end{align*} but since the prior is a Uniform on $$(-20,40)^p$$, the posterior is a truncated Normal and hence the posterior is then a truncated Normal distribution.