Is there theoretical guarantee that principal components are orthogonal? In the sources I've seen, the support for principal components being orthogonal is rooted in the covariance matrix being symmetrical, which means eigenvectors of the covariance matrix corresponding to distinct eigenvalues are pairwise orthogonal. But from what I can see, there is no theoretical guarantee that all the eigenvalues will be distinct. If we have duplicate values, we may lose the orthogonality unless you form an orthogonal eigenbasis, but I don't think I've seen the concept of "orthogonal eigenbasis" in introductory PCA texts where it is taught that "PCs are orthogonal."
So it seems the concept of "principal components" being "orthogonal" is more from a practical perspective because we probably don't see many cases where we see duplicate eigenvalues for a covariance matrix. Is this the case, or is there actually some theoretical guarantee that we'd get distinct eigenvalues for the covariance matrix? I can't see the latter being true since we could run into the case of rank deficiency, and have multiple eigenvalues be zero.
 A: What you are after here, I think, is diagonalization.
All matrices have eigenvectors. Eigenvectors that correspond to different eigenvalues are often orthogonal, or at least linearly independent.
Eigenvectors that correspond to the same eigenvalue can be made orthogonal, i.e. given matrix $\mathbf{A}$ and two eigenvectors $\mathbf{v}_1\neq\mathbf{v}_2$ such that
$$
\mathbf{A}.\mathbf{v}_{1,2}=\lambda \mathbf{v}_{1,2}
$$
For some eigenvalue $\lambda$, note that for any constants $\alpha,\,\beta$ the vector $\mathbf{w}=\alpha\mathbf{v_1}+\beta\mathbf{v_2}$ is also an eigenvector ($\mathbf{A}.\mathbf{w}=\lambda\mathbf{w}$). Therefore if you choose:
$$
\mathbf{w}=\mathbf{v}_2-\left(\frac{\mathbf{v_1}.\mathbf{v}_2}{\mathbf{v_1}.\mathbf{v_1}}\right)\mathbf{v}_1
$$
You will find that set of these eigenvectors is orthogonal $\{\mathbf{v_1},\,\mathbf{w}\}$ whilst still having eigenvalue of $\lambda$. Above the dot-product is assumed to be the appropriate for the (inner product) vector space of interest.
The problem arises when one does not have enough eigenvectors, i.e. it is possible to have just 2 distinct eigenvectors for a $3\times 3$ matrix. Matrices that do not have such problems, i.e. matrices that always have enough eigenvectors to span the space are called diagonalizable because they can be written as
$$
\mathbf{A}=\mathbf{U}^\dagger.\mathbf{D}.\mathbf{U}
$$
Where $\mathbf{U}$ is unitary and $\mathbf{D}$ is diagonal. One can prove that it is sufficient for matrix to be normal to be diagonalizable. Normal matrix is the one that commutes with its adjoint. For real-valued matrices, with real-valued eigen-values and eigen-vectors this boils down to commuting with its transpose, i.e. if real-valued $n\times n$ matrix   ($\mathbf{A}$) that operates on real-valued vectors satisfies:
$$
\left[\mathbf{A}^T,\,\mathbf{A}\right]=\mathbf{0}
$$
then Spectral Theorem applies, and the matrix can be diagonalized, hence one can always find a set of $n$  orthogonal eigenvectors.
Covariance matrix, being symmetric, trivially satisfies this requirement, so you are guaranteed a complete set of orthogonal eigenvectors.
