4
$\begingroup$

Based on this question: How to build a Bayesian regression model of a response that is a Gaussian mixture

Consider the mixture of normal, $$y_j\sim (N(0,\sigma_1))^{\pi}(N(0, \sigma_2))^{1-\pi}, j=1,2,3,4.$$

My question is what is the conditional distribution of $y_j|\sigma_1^2, \sigma_2^2, z$.

$\endgroup$
  • 1
    $\begingroup$ If I understand you correctly, then there is something wrong if H is four dimensional while Y is 100 dimensional. $\endgroup$ – Benjamin Christoffersen Nov 30 '20 at 8:52
  • 1
    $\begingroup$ @BenjaminChristoffersen Sorry, that is 100. $\endgroup$ – user261225 Nov 30 '20 at 8:57
3
$\begingroup$

Note: In this simplified linear model, the OLS estimator $\hat\beta(y)$ is a sufficient statistic, meaning that the posterior on the parameters is the same given $y$ and given $\hat\beta(y)$.

enter image description here enter image description here

Left graph is a (directed acyclic) graph representing the dependence structure in the model. Right graph is the so-called moral graph associated with it (where parents are linked). It is most useful to find conditional dependencies for building a Gibbs sampler, as a node is independent of everything else given its neighbours, i.e. parents and children. For instance, $\beta$ only depends on $y$, $z$, $X$, and $\sigma=(\sigma_1,\sigma_2)$, but not on $\pi$. $$ \beta| z, \sigma_1, \sigma_2, y\sim f(\beta| z, \sigma_1, \sigma_2,y)\propto f(\beta| z, \sigma_1, \sigma_2)\times f(y|,\beta,X) $$ Similarly, $z$ only depends on $\pi$, $\sigma$, and $\beta$, and not on $y$. And at last $\pi$ solely depends on $z$,$$f(\pi|z,\ldots,y)=f(\pi|z)$$

When considering the full conditional of one component of $\beta$, like $\beta_1$, the density satisfies $$f(\beta_1|\beta_{-1},z, \sigma_1, \sigma_2, y)\sim f(\beta_1| z, \sigma_1, \sigma_2,y)\propto f(\beta| z, \sigma_1, \sigma_2,y)$$ which only depends on $z_1$ (and not $z_2,z_3,z_4$): $$f(\beta_1|\beta_{-1},z, \sigma_1, \sigma_2, y)\sim f(\beta_1| z, \sigma_1, \sigma_2,y)\propto f(\beta_1| z_1, \sigma_1, \sigma_2)\times f(y|X,\beta)$$

Although this should be considered as a separate question, here are the details when running a full conditional Gibbs sampler on $\beta$:

At step 0, start with an arbitrary vector $\beta^{(0)}$ (for instance, the OLS $\hat\beta(y)$, and $\pi^{(0)}$, and generate $z^{(0)}$ from its full conditional distribution.

At step t, given the current state $\beta^{(t)},\sigma^{(t)},z^{(t)},\pi^{(t)}$ of the parameter, do

  1. update $\beta_1^{(t)}$ into $\beta_1^{(t+1)}$ by simulating from $$f(\beta_1|\beta_2^{(t)},\beta_3^{(t)},\beta_4^{(t)}, z^{(t)}, \sigma^{(t)},y)\propto f(\beta_1|z_1^{(t)}, \sigma^{(t)})\times f(y|\beta_1,\beta_2^{(t)},\beta_3^{(t)},\beta_4^{(t)})$$
  2. update $\beta_2^{(t)}$ into $\beta_2^{(t+1)}$ by simulating from $$f(\beta_2|\beta_1^{(t+1)},\beta_3^{(t)},\beta_4^{(t)}, z^{(t)}, \sigma^{(t)},y)\propto f(\beta_2|z_2^{(t)}, \sigma^{(t)})\times f(y|\beta_1^{(t+1)},\beta_2,\beta_3^{(t)},\beta_4^{(t)})$$
  3. update $\beta_3^{(t)}$ into $\beta_3^{(t+1)}$ by simulating from $$f(\beta_3|\beta_1^{(t+1)},\beta_2^{(t+1)},\beta_4^{(t)}, z^{(t)}, \sigma^{(t)},y)\propto f(\beta_3|z_3^{(t)}, \sigma^{(t)})\times f(y|\beta_1^{(t+1)},\beta_2^{(t+1)},\beta_3,\beta_4^{(t)})$$
  4. update $\beta_4^{(t)}$ into $\beta_4^{(t+1)}$ by simulating from $$f(\beta_4|\beta_1^{(t+1)},\beta_2^{(t+1)},\beta_3^{(t+1)}, z^{(t)}, \sigma^{(t)},y)\propto f(\beta_4|z_4^{(t)}, \sigma^{(t)})\times f(y|\beta_1^{(t+1)},\beta_2^{(t+1)},\beta_3^{(t+1)},\beta_4)$$
$\endgroup$
  • $\begingroup$ Thank you very much! $\endgroup$ – user261225 Nov 30 '20 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy