3
$\begingroup$

I was looking at the formula for the conditional covariance of a partitioned matrix. I understand the intuition behind the equation for the conditional covariance, but I'm not sure how to show that the covariance matrix is positive definite.

$$\text{For }\bf{x} \sim N(\bf{\mu},\Sigma),$$

$$ \bf{x}= \begin{bmatrix} \bf{x}_a \\ \bf{x}_b \end{bmatrix} $$

$$ \bf{\mu}= \begin{bmatrix} \bf{\mu}_a \\ \bf{\mu}_b \end{bmatrix} $$

$$ \Sigma = \begin{bmatrix} \Sigma _{aa} & \Sigma _{ab} \\ \Sigma _{ba} & \Sigma _{bb} \end{bmatrix} $$

$$ \bf{x}_a|\bf{x}_b \sim N(\bf{\mu}_{a|b},\Sigma_{a|b}) $$ where

$$ \bf{\mu}_{a|b}=\bf{\mu}_a+\Sigma _{ab}\Sigma^{-1}_{bb}(\bf{x}_b-\bf{x}_a) $$ $$ \Sigma_{a|b}=\Sigma_{aa}-\Sigma_{ab}\Sigma^{-1}_{bb}\Sigma_{ba} $$

$\endgroup$
2
$\begingroup$

Elsewhere in mathematics, this procedure is called completing the square. As is familiar from the Quadratic Formula, it amounts to a linear change of variable. The new variables to use are $({\bf x}_a', {\bf y}')$ = $({\bf x}_a', ({\bf x}_b + \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a)')$, because

$$\left( \begin{array}{cc} {\bf x}_a' & {\bf y}' \end{array} \right) \left( \begin{array}{cc} \Sigma_{a|b} & 0 \\ 0 & \Sigma_{bb} \end{array} \right) \left( \begin{array}{c} {\bf x}_a \\ {\bf y} \end{array} \right) = {\bf x}_a' \Sigma_{a|b} {\bf x}_a +{\bf y}'\Sigma_{bb} {\bf y} $$

with

$${\bf x}_a' \Sigma_{a|b} {\bf x}_a = {\bf x}_a' \left(\Sigma_{aa}-\Sigma_{ab}\Sigma^{-1}_{bb}\Sigma_{ba}\right) {\bf x}_a = {\bf x}_a' \Sigma_{aa} {\bf x}_a - {\bf x}_a' \Sigma_{ab}\Sigma^{-1}_{bb}\Sigma_{ba} \bf{x}_a$$

and

$${\bf y}'\Sigma_{bb}{\bf y} = ({\bf x}_b + \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a)'\Sigma_{bb} ({\bf x}_b + \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a).$$

The latter is simplified by the symmetry of the covariance matrix, $\Sigma_{ba}' = \Sigma_{ab}$ and $\Sigma_{bb}' = \Sigma_{bb}$:

$$={\bf x}_b'\Sigma_{bb} {\bf x}_b + {\bf x}_b'\Sigma_{bb}\left(\Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a\right) + \left( \Sigma^{-1}_{bb}\Sigma_{ba}\bf{x}_a\right)'\Sigma_{bb}{\bf x}_b + \left( \Sigma^{-1}_{bb}\Sigma_{ba}\bf{x}_a\right)'\Sigma_{bb}\left(\Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a\right) \\ ={\bf x}_b' \Sigma_{bb} {\bf x}_b +{\bf x}_b' \Sigma_{ba} {\bf x}_a + {\bf x}_a' \Sigma_{ba}' (\Sigma_{bb}^{-1})' \Sigma_{bb}{\bf x}_b + {\bf x}_a' \Sigma_{ba}' (\Sigma_{bb}^{-1})' \Sigma_{bb} \left(\Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a\right) \\ ={\bf x}_b' \Sigma_{bb} {\bf x}_b + {\bf x}_b' \Sigma_{ba} {\bf x}_a + {\bf x}_a' \Sigma_{ab}{\bf x}_b + {\bf x}_a' \Sigma_{ab} \Sigma^{-1}_{bb}\Sigma_{ba}{\bf x}_a.$$

Adding up,

$${\bf x}_a' \Sigma_{a|b} {\bf x}_a + {\bf y}'\Sigma_{bb} {\bf y} = {\bf x}_a' \Sigma_{aa} {\bf x}_a + {\bf x}_b' \Sigma_{bb} {\bf x}_b + {\bf x}_b' \Sigma_{ba} {\bf x}_a + {\bf x}_a' \Sigma_{ab} {\bf x}_b\\ = \left( \begin{array}{cc} {\bf x}_a' & {\bf x}_b' \end{array} \right) \left( \begin{array}{cc} \Sigma_{aa} & \Sigma_{ab} \\ \Sigma_{ba} & \Sigma_{bb} \end{array} \right) \left( \begin{array}{c} {\bf x}_a \\ {\bf x}_b \end{array} \right). $$

By considering the possibilities ${\bf y}=0$ and ${\bf x}_a=0$, it follows immediately that both $\Sigma_{a|b}$ and $\Sigma_{bb}$ (respectively) are positive definite.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.