# How to get the eigenvalue expansion of the covariance matrix?

Working through Bishops’s Pattern Recognition and Machine Learning and have the following question regarding the Eigenvalue expansion of a covariance matrix:

“ Assume we have a symmetric real-valued covariance matrix $$\mathbb\Sigma$$ for a random vector $$x \in \mathbb{R}^D$$.

Consider the eigenvector equation for this matrix: $$\Sigma \mathbf{u_i}=\lambda_i\mathbf{u}_i$$ where $$i=1,...,D$$

As $$\Sigma$$ Is real and symmetric, its eigenvalues will be real and its eigenvectors can be chosen to form an orthonormal set so that $$\mathbf{u}_i^T\mathbf{u_j} = I_{ij}$$ Where $$I_{ij}$$ is the ij-the entry of the identity matrix.

The covariance matrix can be expressed as an expansion in terms of its eigenvectors in the form $$\Sigma=\sum^D_{i=1}\lambda_i\mathbf{u}_i\mathbf{u_i}^T$$

Why is the last statement true? At the moment it looks to me as if $$\Sigma$$ Is being assumed to be a equal to the matrix of eigenvalues but I don’t think this is legitimate... I think I’m missing something at the moment

(My best guess is the following: Is it the eigenvalue decomposition of $$\Sigma$$ Is $$\Sigma=U^T\Lambda U$$ where U is an orthogonal matrix whose columns are the eigenvectors of $$\Sigma$$, $$\Lambda$$ Is the corresponding diagonal matrix of eigenvectors.

But if ,as they’ve said, $$\mathbf{u}_i^T\mathbf{u_j} = I_{ij}$$ and $$\mathbf{u_i}$$ are the columns of U, Are they implying that the columns of U can be permuted so that $$U_{\text{[permuted}}=I$$ and thus $$\Sigma =\Lambda$$

This doesn’t seem right to me...)

• This is called the Spectral Theorem, which you can search under that name (or look up in the index of most linear algebra textbooks).
– whuber
Nov 6, 2020 at 12:42

Your intuition on taking the diagonalization of $$\Sigma$$ is correct; since covariance matrices are symmetric, they are always diagonalizable, and furthermore $$U$$ is an orthogonal matrix. This is a direct consequence of the spectral theorem for symmetric matrices.
The summation that your question is about simply comes down to writing the diagonalization of $$\Sigma$$ in summation form.
Furthermore, you are correct in your assertion that the columns of $$U$$ can be permuted (with appropriate permutations of $$\Sigma$$ as well). However, I don't quite follow how you end up with $$U_{[permuted]} = I$$. $$\Lambda = \Sigma$$ is certainly not true in general. While $$U^\top U = I$$, this doesn't mean that $$U^\top \Lambda U = \Lambda$$, as matrix multiplication is not always commutative.