6
$\begingroup$

Why applying the covariance matrix to the original vector will turn the vector to the direction of largest variance? How do I intuitively understand this?

And what does the inverse of the covariance matrix do? What happens when applying the inverse of covariance matrix to a vector repeatedly?

$\endgroup$
11
$\begingroup$

This is not true for all (non-zero) vectors, but let's explore. The covariance matrix $A$ has an orthonormal basis $v_1, \ldots, v_n$ of eigenvectors with eigenvalues $\lambda_1 \geq \lambda_2 \geq \ldots \geq \lambda_n \geq 0$ (all are non-negative since these correspond to variances). Suppose that $\lambda_1 > \lambda_2$ and take some vector $v = c_1v_1+\ldots+c_nv_n$ where $c_1 > 0$. Then $$A^m v = c_1 \lambda_1^m v_1 + \ldots + c_n \lambda_n^m v_n$$ The term $\lambda_1^m$ dominates the others so $A^mv$ points more and more in the direction of $v_1$. To be more precise let $\alpha$ be the angle between $v_1$ and $A^m v$. Then $$\cos(\alpha) = \frac{\langle v_1, A^mv\rangle}{\lVert A^m v\rVert} = \frac{c_1 \lambda_1^m}{\left(c_1^2\lambda_1^{2m} + \ldots + c_n^2 \lambda_n^{2m}\right)^{\tfrac12}}$$ and this tends to $1$ for increasing power $m$. If we take $c_1<0$ then $\cos(\alpha)$ tends to $-1$ so $A^m v$ points in the direction of $-v_1$ in this case.

Note that if $v$ was taken orthogonal to $v_1$ (so $c_1=0$) then $A^m v$ remains orthogonal to $v_1$.

If $A$ is non-singular (so $\lambda_n >0$) then $v_1, \ldots, v_n$ are still eigenvectors of $A^{-1}$ but now with eigenvalues $0 < \lambda_1^{-1} \leq \lambda_2^{-1} \leq \ldots \leq \lambda_n^{-1}$. Note that the ordering of the eigenvalues reverses and the same story can be applied to $A^{-m} v$. In particular if $\lambda_{n-1}^{-1} < \lambda_n^{-1}$ and the coefficient $c_n$ in $v$ is not zero then $A^{-m}v$ points more and more in the direction of $\pm v_n$.

$\endgroup$
  • $\begingroup$ +1 Excellent answer. I will only add that this is how power iteration works. $\endgroup$ – amoeba Jan 31 '17 at 23:43
  • $\begingroup$ Execellent but perhaps it would be useful to change the notation: you are using "v" for both the eigenvector(s) and a general vector, and this may lead to confusion. $\endgroup$ – MadHatter Jun 1 '17 at 10:53
  • $\begingroup$ @amoeba I understand everything about this answer, except for one thing. How do we know that the eigenvector of the covariance matrix with the biggest eigenvalue is the direction of maximum variance, to begin with? Is there a good link you know of pherhaps? Thanks! $\endgroup$ – Joshua Ronis May 31 at 18:53
  • $\begingroup$ @JoshuaRonis check out youtu.be/cIE2MDxyf80?list=PLBv09BD7ez_5_yapAg86Od6JeeypkS4YM $\endgroup$ – lo tolmencre Jun 8 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.