106
$\begingroup$

We have a multivariate normal vector ${\boldsymbol Y} \sim \mathcal{N}(\boldsymbol\mu, \Sigma)$. Consider partitioning $\boldsymbol\mu$ and ${\boldsymbol Y}$ into $$\boldsymbol\mu = \begin{bmatrix} \boldsymbol\mu_1 \\ \boldsymbol\mu_2 \end{bmatrix} $$ $${\boldsymbol Y}=\begin{bmatrix}{\boldsymbol y}_1 \\ {\boldsymbol y}_2 \end{bmatrix}$$

with a similar partition of $\Sigma$ into $$ \begin{bmatrix} \Sigma_{11} & \Sigma_{12}\\ \Sigma_{21} & \Sigma_{22} \end{bmatrix} $$ Then, $({\boldsymbol y}_1|{\boldsymbol y}_2={\boldsymbol a})$, the conditional distribution of the first partition given the second, is $\mathcal{N}(\overline{\boldsymbol\mu},\overline{\Sigma})$, with mean
$$ \overline{\boldsymbol\mu}=\boldsymbol\mu_1+\Sigma_{12}{\Sigma_{22}}^{-1}({\boldsymbol a}-\boldsymbol\mu_2) $$ and covariance matrix $$ \overline{\Sigma}=\Sigma_{11}-\Sigma_{12}{\Sigma_{22}}^{-1}\Sigma_{21}$$

Actually these results are provided in Wikipedia too, but I have no idea how the $\overline{\boldsymbol\mu}$ and $\overline{\Sigma}$ is derived. These results are crucial, since they are important statistical formula for deriving Kalman filters. Would anyone provide me a derivation steps of deriving $\overline{\boldsymbol\mu}$ and $\overline{\Sigma}$ ? Thank you very much!

$\endgroup$
  • 21
    $\begingroup$ The idea is to use the definition of conditional density $f(y_1\vert y_2=a)=\dfrac{f_{Y_1,Y_2}(y_1,a)}{f_{Y_2}(a)}$. You know that the joint $f_{Y_1,Y_2}$ is a bivariate normal and that the marginal $f_{Y_2}$ is a normal then you just have to replace the values and do the unpleasant algebra. These notes might be of some help. Here is the full proof. $\endgroup$ – user10525 Jun 16 '12 at 18:16
  • 1
    $\begingroup$ Your second link answers the question (+1). Why not put it as an answer @Procrastinator? $\endgroup$ – gui11aume Jun 16 '12 at 22:54
  • 1
    $\begingroup$ I hadn't realized it, but I think I was implicitly using this equation in a conditional PCA. The conditional PCA requires a transformation $\left(I-A'\left(AA'\right)^{-1}A\right)\Sigma$ that is effectively calculating the conditional covariance matrix given some choice of A. $\endgroup$ – John Jul 2 '12 at 15:49
  • $\begingroup$ @Procrastinator - your approach actually requires the knowledge of the Woodbury matrix identity, and the knowledge of block-wise matrix inversion. These result in unnecessarily complicated matrix algebra. $\endgroup$ – probabilityislogic Jul 2 '12 at 16:17
  • 2
    $\begingroup$ @probabilityislogic Actually the result is proved in the link I provided. But it is respectable if you find it more complicated than other methods. In addition, I was not attempting to provide an optimal solution in my comment. Also, my comment was previous to Macro's answer (which I upvoted as you can see). $\endgroup$ – user10525 Jul 2 '12 at 16:25
103
$\begingroup$

You can prove it by explicitly calculating the conditional density by brute force, as in Procrastinator's link (+1) in the comments. But, there's also a theorem that says all conditional distributions of a multivariate normal distribution are normal. Therefore, all that's left is to calculate the mean vector and covariance matrix. I remember we derived this in a time series class in college by cleverly defining a third variable and using its properties to derive the result more simply than the brute force solution in the link (as long as you're comfortable with matrix algebra). I'm going from memory but it was something like this:


Let ${\bf x}_{1}$ be the first partition and ${\bf x}_2$ the second. Now define ${\bf z} = {\bf x}_1 + {\bf A} {\bf x}_2 $ where ${\bf A} = -\Sigma_{12} \Sigma^{-1}_{22}$. Now we can write

\begin{align*} {\rm cov}({\bf z}, {\bf x}_2) &= {\rm cov}( {\bf x}_{1}, {\bf x}_2 ) + {\rm cov}({\bf A}{\bf x}_2, {\bf x}_2) \\ &= \Sigma_{12} + {\bf A} {\rm var}({\bf x}_2) \\ &= \Sigma_{12} - \Sigma_{12} \Sigma^{-1}_{22} \Sigma_{22} \\ &= 0 \end{align*}

Therefore ${\bf z}$ and ${\bf x}_2$ are uncorrelated and, since they are jointly normal, they are independent. Now, clearly $E({\bf z}) = {\boldsymbol \mu}_1 + {\bf A} {\boldsymbol \mu}_2$, therefore it follows that

\begin{align*} E({\bf x}_1 | {\bf x}_2) &= E( {\bf z} - {\bf A} {\bf x}_2 | {\bf x}_2) \\ & = E({\bf z}|{\bf x}_2) - E({\bf A}{\bf x}_2|{\bf x}_2) \\ & = E({\bf z}) - {\bf A}{\bf x}_2 \\ & = {\boldsymbol \mu}_1 + {\bf A} ({\boldsymbol \mu}_2 - {\bf x}_2) \\ & = {\boldsymbol \mu}_1 + \Sigma_{12} \Sigma^{-1}_{22} ({\bf x}_2- {\boldsymbol \mu}_2) \end{align*}

which proves the first part. For the covariance matrix, note that

\begin{align*} {\rm var}({\bf x}_1|{\bf x}_2) &= {\rm var}({\bf z} - {\bf A} {\bf x}_2 | {\bf x}_2) \\ &= {\rm var}({\bf z}|{\bf x}_2) + {\rm var}({\bf A} {\bf x}_2 | {\bf x}_2) - {\bf A}{\rm cov}({\bf z}, -{\bf x}_2) - {\rm cov}({\bf z}, -{\bf x}_2) {\bf A}' \\ &= {\rm var}({\bf z}|{\bf x}_2) \\ &= {\rm var}({\bf z}) \end{align*}

Now we're almost done:

\begin{align*} {\rm var}({\bf x}_1|{\bf x}_2) = {\rm var}( {\bf z} ) &= {\rm var}( {\bf x}_1 + {\bf A} {\bf x}_2 ) \\ &= {\rm var}( {\bf x}_1 ) + {\bf A} {\rm var}( {\bf x}_2 ) {\bf A}' + {\bf A} {\rm cov}({\bf x}_1,{\bf x}_2) + {\rm cov}({\bf x}_2,{\bf x}_1) {\bf A}' \\ &= \Sigma_{11} +\Sigma_{12} \Sigma^{-1}_{22} \Sigma_{22}\Sigma^{-1}_{22}\Sigma_{21} - 2 \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21} \\ &= \Sigma_{11} +\Sigma_{12} \Sigma^{-1}_{22}\Sigma_{21} - 2 \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21} \\ &= \Sigma_{11} -\Sigma_{12} \Sigma^{-1}_{22}\Sigma_{21} \end{align*}

which proves the second part.

Note: For those not very familiar with the matrix algebra used here, this is an excellent resource.

Edit: One property used here this is not in the matrix cookbook (good catch @FlyingPig) is property 6 on the wikipedia page about covariance matrices: which is that for two random vectors $\bf x, y$, $${\rm var}({\bf x}+{\bf y}) = {\rm var}({\bf x})+{\rm var}({\bf y}) + {\rm cov}({\bf x},{\bf y}) + {\rm cov}({\bf y},{\bf x})$$ For scalars, of course, ${\rm cov}(X,Y)={\rm cov}(Y,X)$ but for vectors they are different insofar as the matrices are arranged differently.

$\endgroup$
  • $\begingroup$ Thanks for this brilliant method! There is one matrix algebra does not seem familiar to me, where can I find the formula for opening $var(x_1+Ax_2)$? I haven't found it on the link you sent. $\endgroup$ – Flying pig Jun 17 '12 at 6:35
  • $\begingroup$ @Flyingpig, you're welcome. I believe this is a result of equations $(291),(292)$, combined with an additional property of the variance of the sum of random vectors not written in the Matrix Cookbook - I've added this fact to my answer - thanks for catching that! $\endgroup$ – Macro Jun 17 '12 at 15:02
  • 12
    $\begingroup$ This is a very good answer (+1), but could be improved in terms of the ordering of the approach. We start with saying we want a linear combination $z=Cx=C_1x_1+C_2x_2$ of the whole vector that is independent/uncorrelated with $x_2$. This is because we can use the fact that $p(z|x_2)=p(z)$ which means $var(z|x_2)=var(z)$ and $E(z|x_2)=E(z)$. These in turn lead to expressions for $var(C_1x_1|x_2)$ and $E(C_1x_1|x_2)$. This means we should take $C_1=I$. Now we require $cov(z,x_2)=\Sigma_{12}+C_2\Sigma_{22}=0$. If $\Sigma_{22}$ is invertible we then have $C_2=-\Sigma_{12}\Sigma_{22}^{-1}$. $\endgroup$ – probabilityislogic Jul 2 '12 at 16:00
  • 1
    $\begingroup$ @jakeoung - it is not proving that $C_1=I$, it is setting it to this value, so that we get an expression that contains the variables we want to know about. $\endgroup$ – probabilityislogic Jan 14 '18 at 14:40
  • 1
    $\begingroup$ @jakeoung I also don't quite understand that statement. I understand in this way: If $cov(z, x_2)=0$, then $cov(C_1^{-1} z, x_2) = C_1^{-1} cov( z, x_2)=0$. So the value of $C_1$ is somehow an arbitrary scale. So we set $C_1=I$ for simplicity. $\endgroup$ – Ken T May 5 '18 at 16:03
2
$\begingroup$

The answer by Macro is great, but here is an even simpler way that does not require you to use any outside theorem asserting the conditional distribution. It involves writing the Mahanalobis distance in a form that separates the argument variable for the conditioning statement, and then factorising the normal density accordingly.


Rewriting the Mahanalobis distance for a conditional vector: This derivation uses a matrix inversion formula that uses the Schur complement $\boldsymbol{\Sigma}_\text{S} = \boldsymbol{\Sigma}_{11} - \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} \boldsymbol{\Sigma}_{21}$. We first use the blockwise inversion formula to write the inverse-variance matrix as:

$$\begin{equation} \begin{aligned} \boldsymbol{\Sigma}^{-1} = \begin{bmatrix} \boldsymbol{\Sigma}_{11} & \boldsymbol{\Sigma}_{12} \\ \boldsymbol{\Sigma}_{21} & \boldsymbol{\Sigma}_{22} \\ \end{bmatrix}^{-1} = \begin{bmatrix} \boldsymbol{\Sigma}_{11}^* & \boldsymbol{\Sigma}_{12}^* \\ \boldsymbol{\Sigma}_{21}^* & \boldsymbol{\Sigma}_{22}^* \\ \end{bmatrix}, \end{aligned} \end{equation}$$

where:

$$\begin{equation} \begin{aligned} \begin{matrix} \boldsymbol{\Sigma}_{11}^* = \boldsymbol{\Sigma}_\text{S}^{-1} \text{ } \quad \quad \quad \quad & & & & & \boldsymbol{\Sigma}_{12}^* = -\boldsymbol{\Sigma}_\text{S}^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1}, \quad \quad \quad \\[6pt] \boldsymbol{\Sigma}_{21}^* = - \boldsymbol{\Sigma}_{22}^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_\text{S}^{-1} & & & & & \boldsymbol{\Sigma}_{22}^* = \boldsymbol{\Sigma}_{22}^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_\text{S}^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1}. \text{ } \\[6pt] \end{matrix} \end{aligned} \end{equation}$$

Using this formula we can now write the Mahanalobis distance as:

$$\begin{equation} \begin{aligned} (\boldsymbol{y} - \boldsymbol{\mu})^\text{T} \boldsymbol{\Sigma}^{-1} (\boldsymbol{y} - \boldsymbol{\mu}) &= \begin{bmatrix} \boldsymbol{y}_1 - \boldsymbol{\mu}_1 \\ \boldsymbol{y}_2 - \boldsymbol{\mu}_2 \end{bmatrix}^\text{T} \begin{bmatrix} \boldsymbol{\Sigma}_{11}^* & \boldsymbol{\Sigma}_{12}^* \\ \boldsymbol{\Sigma}_{21}^* & \boldsymbol{\Sigma}_{22}^* \\ \end{bmatrix} \begin{bmatrix} \boldsymbol{y}_1 - \boldsymbol{\mu}_1 \\ \boldsymbol{y}_2 - \boldsymbol{\mu}_2 \end{bmatrix} \\[6pt] &= \quad (\boldsymbol{y}_1 - \boldsymbol{\mu}_1)^\text{T} \boldsymbol{\Sigma}_{11}^* (\boldsymbol{y}_1 - \boldsymbol{\mu}_1) + (\boldsymbol{y}_1 - \boldsymbol{\mu}_1)^\text{T} \boldsymbol{\Sigma}_{12}^* (\boldsymbol{y}_2 - \boldsymbol{\mu}_2) \\[6pt] &\quad + (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)^\text{T} \boldsymbol{\Sigma}_{21}^* (\boldsymbol{y}_1 - \boldsymbol{\mu}_1) + (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)^\text{T} \boldsymbol{\Sigma}_{22}^* (\boldsymbol{y}_2 - \boldsymbol{\mu}_2) \\[6pt] &= (\boldsymbol{y}_1 - (\boldsymbol{\mu}_1 + \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)))^\text{T} \boldsymbol{\Sigma}_\text{S}^{-1} (\boldsymbol{y}_1 - (\boldsymbol{\mu}_1 + \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2))) \\[6pt] &= (\boldsymbol{y}_1 - \boldsymbol{\mu}_*)^\text{T} \boldsymbol{\Sigma}_*^{-1} (\boldsymbol{y}_1 - \boldsymbol{\mu}_*), \\[6pt] \end{aligned} \end{equation}$$

where:

$$\begin{equation} \begin{aligned} \boldsymbol{\mu}_* &\equiv \boldsymbol{\mu}_1 + \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2), \\[8pt] \boldsymbol{\Sigma}_* &\equiv \boldsymbol{\Sigma}_{11} - \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} \boldsymbol{\Sigma}_{21}. \\[6pt] \end{aligned} \end{equation}$$

Note that this result is a general result that does not assume normality of the random vectors. It gives a useful way of re-framing the Mahanalobis distance so that it is a quadratic form with respect to only one of the vectors in the decomposition (with the other absorbed into the mean vector and variance matrix).


Deriving the conditional distribution: Now that we have the above form for the Mahanalobis distance, the rest is easy. We have:

$$\begin{equation} \begin{aligned} p(\boldsymbol{y}_1 | \boldsymbol{y}_2, \boldsymbol{\mu}, \boldsymbol{\Sigma}) &\overset{\boldsymbol{y}_1}{\propto} p(\boldsymbol{y}_1 , \boldsymbol{y}_2 | \boldsymbol{\mu}, \boldsymbol{\Sigma}) \\[12pt] &= \text{N}(\boldsymbol{y} | \boldsymbol{\mu}, \boldsymbol{\Sigma}) \\[10pt] &\overset{\boldsymbol{y}_1}{\propto} \exp \Big( - \frac{1}{2} (\boldsymbol{y} - \boldsymbol{\mu})^\text{T} \boldsymbol{\Sigma}^{-1} (\boldsymbol{y} - \boldsymbol{\mu}) \Big) \\[6pt] &= \exp \Big( - \frac{1}{2} (\boldsymbol{y}_1 - \boldsymbol{\mu}_*)^\text{T} \boldsymbol{\Sigma}_*^{-1} (\boldsymbol{y}_1 - \boldsymbol{\mu}_*) \Big) \\[6pt] &\overset{\boldsymbol{y}_1}{\propto}\text{N}(\boldsymbol{y}_1 | \boldsymbol{\mu}_*, \boldsymbol{\Sigma}_*). \\[6pt] \end{aligned} \end{equation}$$

This establishes that the conditional distribution is also multivariate normal, with the specified conditional mean vector and conditional variance matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.