Proof of a PD covariance matrix for conditional Gaussian

Question

I was looking at the formula for the conditional covariance of a partitioned matrix. I understand the intuition behind the equation for the conditional covariance, but I'm not sure how to show that the covariance matrix is positive definite.

$$\text{For }\bf{x} \sim N(\bf{\mu},\Sigma),$$

$$ \bf{x}= \begin{bmatrix} \bf{x}_a \\ \bf{x}_b \end{bmatrix} $$

$$ \bf{\mu}= \begin{bmatrix} \bf{\mu}_a \\ \bf{\mu}_b \end{bmatrix} $$

$$ \Sigma = \begin{bmatrix} \Sigma _{aa} & \Sigma _{ab} \\ \Sigma _{ba} & \Sigma _{bb} \end{bmatrix} $$

$$ \bf{x}_a|\bf{x}_b \sim N(\bf{\mu}_{a|b},\Sigma_{a|b}) $$ where

$$ \bf{\mu}_{a|b}=\bf{\mu}_a+\Sigma _{ab}\Sigma^{-1}_{bb}(\bf{x}_b-\bf{x}_a) $$ $$ \Sigma_{a|b}=\Sigma_{aa}-\Sigma_{ab}\Sigma^{-1}_{bb}\Sigma_{ba} $$

Answer 1

Elsewhere in mathematics, this procedure is called completing the square. As is familiar from the Quadratic Formula, it amounts to a linear change of variable. The new variables to use are $({\bf x}_a', {\bf y}')$ = $({\bf x}_a', ({\bf x}_b + \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a)')$, because

$$\left( \begin{array}{cc} {\bf x}_a' & {\bf y}' \end{array} \right) \left( \begin{array}{cc} \Sigma_{a|b} & 0 \\ 0 & \Sigma_{bb} \end{array} \right) \left( \begin{array}{c} {\bf x}_a \\ {\bf y} \end{array} \right) = {\bf x}_a' \Sigma_{a|b} {\bf x}_a +{\bf y}'\Sigma_{bb} {\bf y} $$

with

$${\bf x}_a' \Sigma_{a|b} {\bf x}_a = {\bf x}_a' \left(\Sigma_{aa}-\Sigma_{ab}\Sigma^{-1}_{bb}\Sigma_{ba}\right) {\bf x}_a = {\bf x}_a' \Sigma_{aa} {\bf x}_a - {\bf x}_a' \Sigma_{ab}\Sigma^{-1}_{bb}\Sigma_{ba} \bf{x}_a$$

and

$${\bf y}'\Sigma_{bb}{\bf y} = ({\bf x}_b + \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a)'\Sigma_{bb} ({\bf x}_b + \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a).$$

The latter is simplified by the symmetry of the covariance matrix, $\Sigma_{ba}' = \Sigma_{ab}$ and $\Sigma_{bb}' = \Sigma_{bb}$:

$$={\bf x}_b'\Sigma_{bb} {\bf x}_b + {\bf x}_b'\Sigma_{bb}\left(\Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a\right) + \left( \Sigma^{-1}_{bb}\Sigma_{ba}\bf{x}_a\right)'\Sigma_{bb}{\bf x}_b + \left( \Sigma^{-1}_{bb}\Sigma_{ba}\bf{x}_a\right)'\Sigma_{bb}\left(\Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a\right) \\ ={\bf x}_b' \Sigma_{bb} {\bf x}_b +{\bf x}_b' \Sigma_{ba} {\bf x}_a + {\bf x}_a' \Sigma_{ba}' (\Sigma_{bb}^{-1})' \Sigma_{bb}{\bf x}_b + {\bf x}_a' \Sigma_{ba}' (\Sigma_{bb}^{-1})' \Sigma_{bb} \left(\Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a\right) \\ ={\bf x}_b' \Sigma_{bb} {\bf x}_b + {\bf x}_b' \Sigma_{ba} {\bf x}_a + {\bf x}_a' \Sigma_{ab}{\bf x}_b + {\bf x}_a' \Sigma_{ab} \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a.$$

Adding up,

$${\bf x}_a' \Sigma_{a|b} {\bf x}_a + {\bf y}'\Sigma_{bb} {\bf y} = {\bf x}_a' \Sigma_{aa} {\bf x}_a + {\bf x}_b' \Sigma_{bb} {\bf x}_b + {\bf x}_b' \Sigma_{ba} {\bf x}_a + {\bf x}_a' \Sigma_{ab} {\bf x}_b\\ = \left( \begin{array}{cc} {\bf x}_a' & {\bf x}_b' \end{array} \right) \left( \begin{array}{cc} \Sigma_{aa} & \Sigma_{ab} \\ \Sigma_{ba} & \Sigma_{bb} \end{array} \right) \left( \begin{array}{c} {\bf x}_a \\ {\bf x}_b \end{array} \right). $$

By considering the possibilities ${\bf y}=0$ and ${\bf x}_a=0$, it follows immediately that both $\Sigma_{a|b}$ and $\Sigma_{bb}$ (respectively) are positive definite.