Skip to main content
3 of 8
Edit to correct error in decomposition
Ben
  • 132.9k
  • 7
  • 255
  • 588

The answer by Macro is great, but here is an even simpler way that does not require you to use any outside theorem asserting the conditional distribution. It involves writing the Mahanalobis distance in a form that separates the argument variable for the conditioning statement, and then factorising the normal density accordingly.


Rewriting the Mahanalobis distance for a conditional vector: This derivation uses a matrix inversion formula that uses the Schur complement $\boldsymbol{\Sigma}_* \equiv \boldsymbol{\Sigma}_{11} - \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} \boldsymbol{\Sigma}_{21}$. We first use the blockwise inversion formula to write the inverse-variance matrix as:

$$\begin{equation} \begin{aligned} \boldsymbol{\Sigma}^{-1} = \begin{bmatrix} \boldsymbol{\Sigma}_{11} & \boldsymbol{\Sigma}_{12} \\ \boldsymbol{\Sigma}_{21} & \boldsymbol{\Sigma}_{22} \\ \end{bmatrix}^{-1} = \begin{bmatrix} \boldsymbol{\Sigma}_{11}^* & \boldsymbol{\Sigma}_{12}^* \\ \boldsymbol{\Sigma}_{21}^* & \boldsymbol{\Sigma}_{22}^* \\ \end{bmatrix}, \end{aligned} \end{equation}$$

where:

$$\begin{equation} \begin{aligned} \begin{matrix} \boldsymbol{\Sigma}_{11}^* = \boldsymbol{\Sigma}_*^{-1} \text{ } \quad \quad \quad \quad & & & & & \boldsymbol{\Sigma}_{12}^* = -\boldsymbol{\Sigma}_*^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1}, \quad \quad \quad \\[6pt] \boldsymbol{\Sigma}_{21}^* = - \boldsymbol{\Sigma}_{22}^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_*^{-1} & & & & & \boldsymbol{\Sigma}_{22}^* = \boldsymbol{\Sigma}_{22}^{-1} + \boldsymbol{\Sigma}_{22}^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_*^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1}. \text{ } \\[6pt] \end{matrix} \end{aligned} \end{equation}$$

Using this formula we can now write the Mahanalobis distance as:

$$\begin{equation} \begin{aligned} (\boldsymbol{y} &- \boldsymbol{\mu})^\text{T} \boldsymbol{\Sigma}^{-1} (\boldsymbol{y} - \boldsymbol{\mu}) \\[6pt] &= \begin{bmatrix} \boldsymbol{y}_1 - \boldsymbol{\mu}_1 \\ \boldsymbol{y}_2 - \boldsymbol{\mu}_2 \end{bmatrix}^\text{T} \begin{bmatrix} \boldsymbol{\Sigma}_{11}^* & \boldsymbol{\Sigma}_{12}^* \\ \boldsymbol{\Sigma}_{21}^* & \boldsymbol{\Sigma}_{22}^* \\ \end{bmatrix} \begin{bmatrix} \boldsymbol{y}_1 - \boldsymbol{\mu}_1 \\ \boldsymbol{y}_2 - \boldsymbol{\mu}_2 \end{bmatrix} \\[6pt] &= \quad (\boldsymbol{y}_1 - \boldsymbol{\mu}_1)^\text{T} \boldsymbol{\Sigma}_{11}^* (\boldsymbol{y}_1 - \boldsymbol{\mu}_1) + (\boldsymbol{y}_1 - \boldsymbol{\mu}_1)^\text{T} \boldsymbol{\Sigma}_{12}^* (\boldsymbol{y}_2 - \boldsymbol{\mu}_2) \\[6pt] &\quad + (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)^\text{T} \boldsymbol{\Sigma}_{21}^* (\boldsymbol{y}_1 - \boldsymbol{\mu}_1) + (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)^\text{T} \boldsymbol{\Sigma}_{22}^* (\boldsymbol{y}_2 - \boldsymbol{\mu}_2) \\[6pt] &= \quad (\boldsymbol{y}_1 - \boldsymbol{\mu}_1)^\text{T} \boldsymbol{\Sigma}_\text{S}^{-1} (\boldsymbol{y}_1 - \boldsymbol{\mu}_1) - (\boldsymbol{y}_1 - \boldsymbol{\mu}_1)^\text{T} \boldsymbol{\Sigma}_\text{S}^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2) \\[6pt] &\quad - (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)^\text{T} \boldsymbol{\Sigma}_{22}^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_\text{S}^{-1} (\boldsymbol{y}_1 - \boldsymbol{\mu}_1) + (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)^\text{T} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2) \\[6pt] &\quad + (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)^\text{T} \boldsymbol{\Sigma}_{22}^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_\text{S}^{-1} \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2) \\[6pt] &= (\boldsymbol{y}_1 - (\boldsymbol{\mu}_1 + \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)))^\text{T} \boldsymbol{\Sigma}_\text{S}^{-1} (\boldsymbol{y}_1 - (\boldsymbol{\mu}_1 + \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2))) \\[6pt] &\quad + (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)^\text{T} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2) \\[6pt] &= (\boldsymbol{y}_1 - \boldsymbol{\mu}_*)^\text{T} \boldsymbol{\Sigma}_*^{-1} (\boldsymbol{y}_1 - \boldsymbol{\mu}_*) + (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)^\text{T} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2) , \\[6pt] \end{aligned} \end{equation}$$

where $\boldsymbol{\mu}_* \equiv \boldsymbol{\mu}_1 + \boldsymbol{\Sigma}_{12} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)$ is the adjusted mean vector. Note that this result is a general result that does not assume normality of the random vectors. It gives a useful way of decomposing the Mahanalobis distance so that it consists of a sum of quadratic forms on the marginal and conditional parts. In the conditional part the conditioning vector $\boldsymbol{y}_2$ is absorbed into the mean vector and variance matrix. To clarify the form, we repeat the equation with labelling of terms:

$$(\boldsymbol{y} - \boldsymbol{\mu})^\text{T} \boldsymbol{\Sigma}^{-1} (\boldsymbol{y} - \boldsymbol{\mu}) = \underbrace{(\boldsymbol{y}_1 - \boldsymbol{\mu}_*)^\text{T} \boldsymbol{\Sigma}_*^{-1} (\boldsymbol{y}_1 - \boldsymbol{\mu}_*)}_\text{Conditional Part} + \underbrace{(\boldsymbol{y}_2 - \boldsymbol{\mu}_2)^\text{T} \boldsymbol{\Sigma}_{22}^{-1} (\boldsymbol{y}_2 - \boldsymbol{\mu}_2)}_\text{Marginal Part}.$$


Deriving the conditional distribution: Now that we have the above form for the Mahanalobis distance, the rest is easy. We have:

$$\begin{equation} \begin{aligned} p(\boldsymbol{y}_1 | \boldsymbol{y}_2, \boldsymbol{\mu}, \boldsymbol{\Sigma}) &\overset{\boldsymbol{y}_1}{\propto} p(\boldsymbol{y}_1 , \boldsymbol{y}_2 | \boldsymbol{\mu}, \boldsymbol{\Sigma}) \\[12pt] &= \text{N}(\boldsymbol{y} | \boldsymbol{\mu}, \boldsymbol{\Sigma}) \\[10pt] &\overset{\boldsymbol{y}_1}{\propto} \exp \Big( - \frac{1}{2} (\boldsymbol{y} - \boldsymbol{\mu})^\text{T} \boldsymbol{\Sigma}^{-1} (\boldsymbol{y} - \boldsymbol{\mu}) \Big) \\[6pt] &\overset{\boldsymbol{y}_1}{\propto} \exp \Big( - \frac{1}{2} (\boldsymbol{y}_1 - \boldsymbol{\mu}_*)^\text{T} \boldsymbol{\Sigma}_*^{-1} (\boldsymbol{y}_1 - \boldsymbol{\mu}_*) \Big) \\[6pt] &\overset{\boldsymbol{y}_1}{\propto}\text{N}(\boldsymbol{y}_1 | \boldsymbol{\mu}_*, \boldsymbol{\Sigma}_*). \\[6pt] \end{aligned} \end{equation}$$

This establishes that the conditional distribution is also multivariate normal, with the specified conditional mean vector and conditional variance matrix.

Ben
  • 132.9k
  • 7
  • 255
  • 588