# is covariance matrix and eigen vector question

is every covariance matrix 's eigenvector is orthogonal? why symmetric matrix's eigenvector is orthogonal? can you show some example? and the reason?

$$\Sigma {\bf U_i} = \lambda_i U_i$$

additional question) why first eigenvalue is the direction of covariance matrix? and second eigenvalue(what is second eigenvalue?) is the 90 degrees of the first eigenvalue?

• This answer on the math stackexchange site may help. Aug 27, 2016 at 15:16

Every covariance matrix is symmetric, that is, $$\Sigma^T=\Sigma$$. Then eigenvectors corresponding to distinct eigenvalues are orthogonal (you left out the important qualifier!). Here is a simple algebraic proof.
Let $$\Sigma=\Sigma^T$$ be symmetric, with eigenvectors $$x, y$$ such that $$\Sigma x=\lambda x, \Sigma y=\mu y, \quad \lambda \not=\mu, \lambda \not=0.$$ $$x$$ and $$y$$ are orthogonal means that $$x^Ty=0$$. Now $$x^Ty = \left( \frac1\lambda \Sigma x\right)^T y = \frac1\lambda x^T \Sigma y =\\ \frac1\lambda \mu x^Ty$$ By subtraction we have $$x^Ty - \frac\mu\lambda x^T y =\left( 1-\frac\mu\lambda\right) x^T y=0$$ and since $$\lambda \not=\mu$$ necessarily $$x^Ty=0$$.
• You don't need to assume $\lambda\ne 0,$ suggesting there's a simpler demonstration. Indeed, the symmetry of $\Sigma$ implies $$\lambda y^\prime x = y^\prime (\Sigma x) = x^\prime\Sigma y = x^\prime \mu y = \mu y^\prime x.$$ Therefore $$(\lambda-\mu)\, y^\prime x = 0,$$ whence $\lambda=\mu$ or $y^\prime x = 0,$ QED. (I see now that this is exactly the approach taken in the answer you link to.) More importantly, it's worth remarking that nonzero eigenvectors with the same eigenvalues are not necessarily orthogonal.