Why applying the covariance matrix to the original vector will turn the vector to the direction of largest variance? How do I intuitively understand this?
And what does the inverse of the covariance matrix do? What happens when applying the inverse of covariance matrix to a vector repeatedly?
This is not true for all (non-zero) vectors, but let's explore. The covariance matrix $A$ has an orthonormal basis $v_1, \ldots, v_n$ of eigenvectors with eigenvalues $\lambda_1 \geq \lambda_2 \geq \ldots \geq \lambda_n \geq 0$ (all are non-negative since these correspond to variances). Suppose that $\lambda_1 > \lambda_2$ and take some vector $v = c_1v_1+\ldots+c_nv_n$ where $c_1 > 0$. Then $$A^m v = c_1 \lambda_1^m v_1 + \ldots + c_n \lambda_n^m v_n$$ The term $\lambda_1^m$ dominates the others so $A^mv$ points more and more in the direction of $v_1$. To be more precise let $\alpha$ be the angle between $v_1$ and $A^m v$. Then $$\cos(\alpha) = \frac{\langle v_1, A^mv\rangle}{\lVert A^m v\rVert} = \frac{c_1 \lambda_1^m}{\left(c_1^2\lambda_1^{2m} + \ldots + c_n^2 \lambda_n^{2m}\right)^{\tfrac12}}$$ and this tends to $1$ for increasing power $m$. If we take $c_1<0$ then $\cos(\alpha)$ tends to $-1$ so $A^m v$ points in the direction of $-v_1$ in this case.
Note that if $v$ was taken orthogonal to $v_1$ (so $c_1=0$) then $A^m v$ remains orthogonal to $v_1$.
If $A$ is non-singular (so $\lambda_n >0$) then $v_1, \ldots, v_n$ are still eigenvectors of $A^{-1}$ but now with eigenvalues $0 < \lambda_1^{-1} \leq \lambda_2^{-1} \leq \ldots \leq \lambda_n^{-1}$. Note that the ordering of the eigenvalues reverses and the same story can be applied to $A^{-m} v$. In particular if $\lambda_{n-1}^{-1} < \lambda_n^{-1}$ and the coefficient $c_n$ in $v$ is not zero then $A^{-m}v$ points more and more in the direction of $\pm v_n$.
