1
$\begingroup$

is every covariance matrix 's eigenvector is orthogonal? why symmetric matrix's eigenvector is orthogonal? can you show some example? and the reason?

$$\Sigma {\bf U_i} = \lambda_i U_i$$

additional question) why first eigenvalue is the direction of covariance matrix? and second eigenvalue(what is second eigenvalue?) is the 90 degrees of the first eigenvalue?

$\endgroup$
1
  • 2
    $\begingroup$ This answer on the math stackexchange site may help. $\endgroup$
    – GeoMatt22
    Aug 27, 2016 at 15:16

1 Answer 1

0
$\begingroup$

Every covariance matrix is symmetric, that is, $\Sigma^T=\Sigma$. Then eigenvectors corresponding to distinct eigenvalues are orthogonal (you left out the important qualifier!). Here is a simple algebraic proof.

Let $\Sigma=\Sigma^T$ be symmetric, with eigenvectors $x, y$ such that $$ \Sigma x=\lambda x, \Sigma y=\mu y, \quad \lambda \not=\mu, \lambda \not=0. $$ $x$ and $y$ are orthogonal means that $x^Ty=0$. Now $$ x^Ty = \left( \frac1\lambda \Sigma x\right)^T y = \frac1\lambda x^T \Sigma y =\\ \frac1\lambda \mu x^Ty $$ By subtraction we have $$ x^Ty - \frac\mu\lambda x^T y =\left( 1-\frac\mu\lambda\right) x^T y=0 $$ and since $\lambda \not=\mu$ necessarily $x^Ty=0$.

That is the standard proof, and more details is at https://math.stackexchange.com/questions/82467/eigenvectors-of-real-symmetric-matrices-are-orthogonal

For your second question, what do you mean by direction of covariance matrix?

$\endgroup$
1
  • $\begingroup$ You don't need to assume $\lambda\ne 0,$ suggesting there's a simpler demonstration. Indeed, the symmetry of $\Sigma$ implies $$\lambda y^\prime x = y^\prime (\Sigma x) = x^\prime\Sigma y = x^\prime \mu y = \mu y^\prime x.$$ Therefore $$(\lambda-\mu)\, y^\prime x = 0,$$ whence $\lambda=\mu$ or $y^\prime x = 0,$ QED. (I see now that this is exactly the approach taken in the answer you link to.) More importantly, it's worth remarking that nonzero eigenvectors with the same eigenvalues are not necessarily orthogonal. $\endgroup$
    – whuber
    Dec 18, 2021 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.