Deriving full conditionals from joint distributions?

Question

In this link (https://www.youtube.com/watch?v=a_08GKWHFWo), the author derives the conditional distributions from the joint; but I got lost in the mechanics of what happened, the process was overly simplified. The topic of the video was on Gibbs Sampling, the derivation of full conditionals was not the subject but a supporting part of the video.

When I try to prove that his conditional distributions are accurate and I write the full joint PDF, divided a single variable PDF, I get a messy equation that is not easy to simplify.

$$\theta ~ N_2\left(\mu= \begin{bmatrix} 0\\0\end{bmatrix}, cov = \begin{bmatrix}1 &0.9 \\ 0.9 & 1 \end{bmatrix}\right)$$ $$\theta_1|\theta_2 \sim N(\mu= 0.9\theta_2, \sigma= 1-0.9^2)$$ $$\theta_2|\theta_1 \sim N(\mu= 0.9\theta_1, \sigma=1-0.9^2)$$

Above is pretty much all he shows, but I'm completely lost on how we've gone from joint to conditional(s). Am I missing something obvious?

Is there a rule/process for deriving these conditional distributions (with Gibbs application in mind)?

Answer 1

For multivariate normal distribution, it is known that

$$\begin{bmatrix}\theta_1 \\ \theta_2 \end{bmatrix} \sim N \left(\begin{bmatrix} \mu_1 \\ \mu_2\end{bmatrix}, \begin{bmatrix} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22}\end{bmatrix} \right)$$

then we have $$\theta_1|\theta_2 \sim N\left( \mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(\theta_2 - \mu_2), \Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\right).$$

This is a standard result. A proof can be found here.

The idea of proof is we first compute the ratio of densities from definition and identify that it indeed remains a gaussian distribution (i.e. the exponent is quadratic form with negative semidefinite matrix). We then compute the corresponding conditional covariance matrix using matrix inversion theorem (or schur complement) and the rest are details.

For your problem, $\mu_1=\mu_2=0$, $\Sigma_{11}=\Sigma_{22}=1$, $\Sigma_{12}=\Sigma_{21}=\rho=0.9$,

the conditional expected value is $$\mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(\theta_2-\mu_2) = 0 + \rho (1^{-1}) (\theta_Y-0)=\rho\theta_2$$

and the covariance matrix is

$$\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}=1-\rho (1^{-1})\rho=1-\rho^2.$$

Answer 2

Theorem: Let $x$ follow a multivariate normal distribution (Definition)

$$ \label{eq:mvn} x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \sim \mathcal{N}\left( \begin{bmatrix} \mu_1 \\ \mu_2 \end{bmatrix}, \begin{bmatrix} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{bmatrix} \right) \; . $$

Then, the conditional distribution (Definition) of any subset vector $x_1$, given the complement vector $x_2$, is also a multivariate normal distribution

$$ \label{eq:mvn-cond} x_1|x_2 \sim \mathcal{N}(\mu_{1|2}, \Sigma_{1|2}) $$

where the conditional mean and covariance are

$$ \label{eq:mvn-cond-hyp} \begin{split} \mu_{1|2} &= \mu_1 + \Sigma_{12} \Sigma_{22}^{-1} (x_2 - \mu_2) \\ \Sigma_{1|2} &= \Sigma_{11} - \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21} \; . \end{split} $$

Proof: This can be shown by combining the joint and the marginal densities of the multivariate normal distribution and then applying Bayes' theorem (see eq. 8 in Proof).