Derivation for the expression for $ \mathbf{x-\mu} $ in a multivariate Gaussian distribution

Question

The multivariate normal distribution is given by

$$\text{N}(\textbf{x}|\mathbf{\mu}, \mathbf{\Sigma}) = (2\pi)^{-D/2} |\mathbf{\Sigma}|^{-1/2} \exp \Bigg( - \frac{1}{2}{(\mathbf{x - \mu})}^\text{T} \mathbf{\Sigma}^{-1}(\mathbf{x-\mu}) \Bigg).$$

Let the eigenvector equation for the covariance matrix be

$$\mathbf{\Sigma} \mathbf{u}_i = \lambda_{i} \mathbf{u}_i.$$

Equation 2.60 of Bishop's book on Pattern Recognition and machine learning reads

$$\mathbf{x} - \boldsymbol{\mu} = \sum_{j=1}^{D} \mathbf{u}_j^\text{T} (\mathbf{x} - \boldsymbol{\mu}) \mathbf{u}_j.$$

Bishop's book states that the above equation can be derived from the eigenvector expansion of the covariance matrix together with the completeness of the set of eigenvectors. My questions are

1. What is completeness of the set of eigenvectors?
2. How does that lead to the derivation of the above equation?

Answer 1

Completeness means that the basis of eigenvectors ${\bf u}_i$ span the entire vector space. So any vector can be expressed as a linear combination of the basis vectors. Since ${\bf x} - {\bf \mu}$ is any vector, completeness implies that we can express it as $$ {\bf x} - {\bf\mu} = \sum_j \alpha_j {\bf u}_j, $$ where $\alpha_j$ is a scalar coefficient.

The eigenvectors of a real symmetric matrix (which ${\bf \Sigma}$ is) are orthogonal, and can be chosen to be orthonormal, which was the case for these values. So ${\bf u}_i^\intercal {\bf u}_j = \delta_{i j}.$

To calculate the coefficients we left multiply both sides of the above equation by ${\bf u}_i^\intercal$:

$$ \begin{split} {\bf u}_i^\intercal \left( {\bf x} - {\bf \mu} \right) &= \sum_j \alpha_j {\bf u}_i^\intercal {\bf u}_j \\ &= \sum_j \alpha_j \delta_{i j} \\ &= \alpha_i \end{split} $$