In the sources I've seen, the support for principal components being orthogonal is rooted in the covariance matrix being symmetrical, which means eigenvectors of the covariance matrix corresponding to distinct eigenvalues are pairwise orthogonal. But from what I can see, there is no theoretical guarantee that all the eigenvalues will be distinct. If we have duplicate values, we may lose the orthogonality unless you form an orthogonal eigenbasis, but I don't think I've seen the concept of "orthogonal eigenbasis" in introductory PCA texts where it is taught that "PCs are orthogonal."

So it seems the concept of "principal components" being "orthogonal" is more from a practical perspective because we probably don't see many cases where we see duplicate eigenvalues for a covariance matrix. Is this the case, or is there actually some theoretical guarantee that we'd get distinct eigenvalues for the covariance matrix? I can't see the latter being true since we could run into the case of rank deficiency, and have multiple eigenvalues be zero.

  • 2
    $\begingroup$ The guarantee remains: it's part of the Spectral Theorem. It's also assured by the standard algorithms to compute PCA, which proceed to adjoin new components that are orthogonal to all previously found ones. Rank deficiency is no problem: that's merely the case where $0$ is an eigenvalue; it may have any multiplicity. $\endgroup$ – whuber May 6 at 16:15
  • 2
    $\begingroup$ Repeated eigenvectors do not prevent orthogonal principal components; they just mean they need not be unique. It is rather like trying to find the three axes of an ellipsoid and then discovering you are dealing with a spheroid: you can still find three orthogonal axes, but there are other possible choices for the orthogonal axes. $\endgroup$ – Henry May 6 at 17:11

What you are after here, I think, is diagonalization.

All matrices have eigenvectors. Eigenvectors that correspond to different eigenvalues are often orthogonal, or at least linearly independent.

Eigenvectors that correspond to the same eigenvalue can be made orthogonal, i.e. given matrix $\mathbf{A}$ and two eigenvectors $\mathbf{v}_1\neq\mathbf{v}_2$ such that

$$ \mathbf{A}.\mathbf{v}_{1,2}=\lambda \mathbf{v}_{1,2} $$

For some eigenvalue $\lambda$, note that for any constants $\alpha,\,\beta$ the vector $\mathbf{w}=\alpha\mathbf{v_1}+\beta\mathbf{v_2}$ is also an eigenvector ($\mathbf{A}.\mathbf{w}=\lambda\mathbf{w}$). Therefore if you choose:

$$ \mathbf{w}=\mathbf{v}_2-\left(\frac{\mathbf{v_1}.\mathbf{v}_2}{\mathbf{v_1}.\mathbf{v_1}}\right)\mathbf{v}_1 $$

You will find that set of these eigenvectors is orthogonal $\{\mathbf{v_1},\,\mathbf{w}\}$ whilst still having eigenvalue of $\lambda$. Above the dot-product is assumed to be the appropriate for the (inner product) vector space of interest.

The problem arises when one does not have enough eigenvectors, i.e. it is possible to have just 2 distinct eigenvectors for a $3\times 3$ matrix. Matrices that do not have such problems, i.e. matrices that always have enough eigenvectors to span the space are called diagonalizable because they can be written as

$$ \mathbf{A}=\mathbf{U}^\dagger.\mathbf{D}.\mathbf{U} $$

Where $\mathbf{U}$ is unitary and $\mathbf{D}$ is diagonal. One can prove that it is sufficient for matrix to be normal to be diagonalizable. Normal matrix is the one that commutes with its adjoint. For real-valued matrices, with real-valued eigen-values and eigen-vectors this boils down to commuting with its transpose, i.e. if real-valued $n\times n$ matrix ($\mathbf{A}$) that operates on real-valued vectors satisfies:

$$ \left[\mathbf{A}^T,\,\mathbf{A}\right]=\mathbf{0} $$

then Spectral Theorem applies, and the matrix can be diagonalized, hence one can always find a set of $n$ orthogonal eigenvectors.

Covariance matrix, being symmetric, trivially satisfies this requirement, so you are guaranteed a complete set of orthogonal eigenvectors.

  • $\begingroup$ Re "All matrices have eigenvectors ... Eigenvectors that correspond to different eigenvalues are orthogonal:" This is incorrect. You need to limit this assertion to real, symmetric matrices. You get it right later, but making this assertion early on might confuse people. $\endgroup$ – whuber May 7 at 14:56
  • $\begingroup$ @whuber, can you please be more specific? Do some matrices have no eigenvectors, or are there matrices for which eigenvectors from different eigenspaces are not orthogonal? $\endgroup$ – Cryo May 7 at 18:25
  • $\begingroup$ @whuber, Axler Linear Algebra Done Right, theorem 5.13, "Suppose $V$ is a complex vector space and $T\in\mathcal{L}(V)$, i.e. a linear operator (matrix). Then $T$ has upper-triangular matrix with respect to some basis of $V$". $\endgroup$ – Cryo May 7 at 19:15
  • 1
    $\begingroup$ The error is in supposing the eigenspaces are generally orthogonal. Note, too, that "there is at least one eigenvalue" presumes you accept complex values. The standard example is a $2\times 2$ rotation matrix, which has two complex conjugate eigenvalues. $\endgroup$ – whuber May 7 at 21:04
  • 1
    $\begingroup$ Happy to presume complex eigenvalues at the beginning and then to specialize to real-valued ones later on. Did clarify the language about eigenvectors, thanks. $\endgroup$ – Cryo May 7 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.