Proof of a PD covariance matrix for conditional Gaussian I was looking at the formula for the conditional covariance of a partitioned matrix. I understand the intuition behind the equation for the conditional covariance, but I'm not sure how to show that the covariance matrix is positive definite.
$$\text{For }\bf{x} \sim N(\bf{\mu},\Sigma),$$
$$
\bf{x}=
\begin{bmatrix} \bf{x}_a \\ \bf{x}_b \end{bmatrix}
$$
$$
\bf{\mu}=
\begin{bmatrix} \bf{\mu}_a \\ \bf{\mu}_b \end{bmatrix}
$$
$$
\Sigma =
\begin{bmatrix} \Sigma _{aa} & \Sigma _{ab} \\ \Sigma _{ba} & \Sigma _{bb} \end{bmatrix}
$$
$$
\bf{x}_a|\bf{x}_b \sim N(\bf{\mu}_{a|b},\Sigma_{a|b})
$$ where
$$
\bf{\mu}_{a|b}=\bf{\mu}_a+\Sigma _{ab}\Sigma^{-1}_{bb}(\bf{x}_b-\bf{x}_a)
$$
$$
\Sigma_{a|b}=\Sigma_{aa}-\Sigma_{ab}\Sigma^{-1}_{bb}\Sigma_{ba}
$$
 A: Elsewhere in mathematics, this procedure is called completing the square.  As is familiar from the Quadratic Formula, it amounts to a linear change of variable. The new variables to use are $({\bf x}_a', {\bf y}')$ = $({\bf x}_a', ({\bf x}_b + \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a)')$, because
$$\left(
\begin{array}{cc}
 {\bf x}_a' & {\bf y}'
\end{array}
\right)
\left(
\begin{array}{cc}
 \Sigma_{a|b} & 0 \\
 0 & \Sigma_{bb}
\end{array}
\right)
\left(
\begin{array}{c}
 {\bf x}_a \\
 {\bf y}
\end{array}
\right) = 
{\bf x}_a' \Sigma_{a|b} {\bf x}_a +{\bf y}'\Sigma_{bb} {\bf y} 
$$
with
$${\bf x}_a' \Sigma_{a|b} {\bf x}_a = {\bf x}_a' \left(\Sigma_{aa}-\Sigma_{ab}\Sigma^{-1}_{bb}\Sigma_{ba}\right) {\bf x}_a
=  {\bf x}_a' \Sigma_{aa} {\bf x}_a - {\bf x}_a' \Sigma_{ab}\Sigma^{-1}_{bb}\Sigma_{ba} \bf{x}_a$$
and
$${\bf y}'\Sigma_{bb}{\bf y} =  ({\bf x}_b + \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a)'\Sigma_{bb}  ({\bf x}_b + \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a).$$
The latter is simplified by the symmetry of the covariance matrix, $\Sigma_{ba}' = \Sigma_{ab}$ and $\Sigma_{bb}' = \Sigma_{bb}$:
$$={\bf x}_b'\Sigma_{bb} {\bf x}_b 
+ {\bf x}_b'\Sigma_{bb}\left(\Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a\right)
+ \left( \Sigma^{-1}_{bb}\Sigma_{ba}\bf{x}_a\right)'\Sigma_{bb}{\bf x}_b 
+ \left( \Sigma^{-1}_{bb}\Sigma_{ba}\bf{x}_a\right)'\Sigma_{bb}\left(\Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a\right) \\
={\bf x}_b' \Sigma_{bb} {\bf x}_b 
+{\bf x}_b' \Sigma_{ba} {\bf x}_a 
+ {\bf x}_a' \Sigma_{ba}' (\Sigma_{bb}^{-1})' \Sigma_{bb}{\bf x}_b
+ {\bf x}_a' \Sigma_{ba}' (\Sigma_{bb}^{-1})' \Sigma_{bb} \left(\Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a\right) \\
={\bf x}_b' \Sigma_{bb} {\bf x}_b 
+ {\bf x}_b' \Sigma_{ba} {\bf x}_a 
+ {\bf x}_a' \Sigma_{ab}{\bf x}_b
+ {\bf x}_a' \Sigma_{ab} \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a.$$
Adding up, 
$${\bf x}_a' \Sigma_{a|b} {\bf x}_a + {\bf y}'\Sigma_{bb} {\bf y} 
= {\bf x}_a' \Sigma_{aa} {\bf x}_a 
+ {\bf x}_b' \Sigma_{bb} {\bf x}_b 
+ {\bf x}_b' \Sigma_{ba} {\bf x}_a 
+ {\bf x}_a' \Sigma_{ab} {\bf x}_b\\
=
\left(
\begin{array}{cc}
 {\bf x}_a' & {\bf x}_b'
\end{array}
\right)
\left(
\begin{array}{cc}
 \Sigma_{aa} & \Sigma_{ab} \\
 \Sigma_{ba} & \Sigma_{bb}
\end{array}
\right)
\left(
\begin{array}{c}
 {\bf x}_a \\
{\bf x}_b
\end{array}
\right).
$$
By considering the possibilities ${\bf y}=0$ and ${\bf x}_a=0$, it follows immediately that both $\Sigma_{a|b}$ and $\Sigma_{bb}$ (respectively) are positive definite.
