Working through Bishops’s Pattern Recognition and Machine Learning and have the following question regarding the Eigenvalue expansion of a covariance matrix:

“ Assume we have a symmetric real-valued covariance matrix $\mathbb\Sigma$ for a random vector $x \in \mathbb{R}^D$.

Consider the eigenvector equation for this matrix: $$\Sigma \mathbf{u_i}=\lambda_i\mathbf{u}_i$$ where $i=1,...,D$

As $\Sigma$ Is real and symmetric, its eigenvalues will be real and its eigenvectors can be chosen to form an orthonormal set so that $$\mathbf{u}_i^T\mathbf{u_j} = I_{ij}$$ Where $I_{ij}$ is the ij-the entry of the identity matrix.

The covariance matrix can be expressed as an expansion in terms of its eigenvectors in the form $$\Sigma=\sum^D_{i=1}\lambda_i\mathbf{u}_i\mathbf{u_i}^T$$

Why is the last statement true? At the moment it looks to me as if $\Sigma$ Is being assumed to be a equal to the matrix of eigenvalues but I don’t think this is legitimate... I think I’m missing something at the moment

(My best guess is the following: Is it the eigenvalue decomposition of $\Sigma$ Is $$\Sigma=U^T\Lambda U$$ where U is an orthogonal matrix whose columns are the eigenvectors of $\Sigma$, $\Lambda$ Is the corresponding diagonal matrix of eigenvectors.

But if ,as they’ve said, $$\mathbf{u}_i^T\mathbf{u_j} = I_{ij}$$ and $\mathbf{u_i}$ are the columns of U, Are they implying that the columns of U can be permuted so that $U_{\text{[permuted}}=I$ and thus $\Sigma =\Lambda$

This doesn’t seem right to me...)

  • $\begingroup$ This is called the Spectral Theorem, which you can search under that name (or look up in the index of most linear algebra textbooks). $\endgroup$ – whuber Nov 6 '20 at 12:42

Your intuition on taking the diagonalization of $\Sigma$ is correct; since covariance matrices are symmetric, they are always diagonalizable, and furthermore $U$ is an orthogonal matrix. This is a direct consequence of the spectral theorem for symmetric matrices.

The summation that your question is about simply comes down to writing the diagonalization of $\Sigma$ in summation form.

Furthermore, you are correct in your assertion that the columns of $U$ can be permuted (with appropriate permutations of $\Sigma$ as well). However, I don't quite follow how you end up with $U_{[permuted]} = I$. $\Lambda = \Sigma$ is certainly not true in general. While $U^\top U = I$, this doesn't mean that $U^\top \Lambda U = \Lambda$, as matrix multiplication is not always commutative.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.