How to get the eigenvalue expansion of the covariance matrix?

Question

Working through Bishops’s Pattern Recognition and Machine Learning and have the following question regarding the Eigenvalue expansion of a covariance matrix:

“ Assume we have a symmetric real-valued covariance matrix $\mathbb\Sigma$ for a random vector $x \in \mathbb{R}^D$.

Consider the eigenvector equation for this matrix: $$\Sigma \mathbf{u_i}=\lambda_i\mathbf{u}_i$$ where $i=1,...,D$

As $\Sigma$ Is real and symmetric, its eigenvalues will be real and its eigenvectors can be chosen to form an orthonormal set so that $$\mathbf{u}_i^T\mathbf{u_j} = I_{ij}$$ Where $I_{ij}$ is the ij-the entry of the identity matrix.

The covariance matrix can be expressed as an expansion in terms of its eigenvectors in the form $$\Sigma=\sum^D_{i=1}\lambda_i\mathbf{u}_i\mathbf{u_i}^T$$”

Why is the last statement true? At the moment it looks to me as if $\Sigma$ Is being assumed to be a equal to the matrix of eigenvalues but I don’t think this is legitimate... I think I’m missing something at the moment

(My best guess is the following: Is it the eigenvalue decomposition of $\Sigma$ Is $$\Sigma=U^T\Lambda U$$ where U is an orthogonal matrix whose columns are the eigenvectors of $\Sigma$, $\Lambda$ Is the corresponding diagonal matrix of eigenvectors.

But if ,as they’ve said, $$\mathbf{u}_i^T\mathbf{u_j} = I_{ij}$$ and $\mathbf{u_i}$ are the columns of U, Are they implying that the columns of U can be permuted so that $U_{\text{[permuted}}=I$ and thus $\Sigma =\Lambda$

This doesn’t seem right to me...)

Answer 1

Your intuition on taking the diagonalization of $\Sigma$ is correct; since covariance matrices are symmetric, they are always diagonalizable, and furthermore $U$ is an orthogonal matrix. This is a direct consequence of the spectral theorem for symmetric matrices.

The summation that your question is about simply comes down to writing the diagonalization of $\Sigma$ in summation form.

Furthermore, you are correct in your assertion that the columns of $U$ can be permuted (with appropriate permutations of $\Sigma$ as well). However, I don't quite follow how you end up with $U_{[permuted]} = I$. $\Lambda = \Sigma$ is certainly not true in general. While $U^\top U = I$, this doesn't mean that $U^\top \Lambda U = \Lambda$, as matrix multiplication is not always commutative.