Suppose ${\bf{X}} = ({X_1},{X_2},\ldots,{X_n})$ are the original components (also random variables) and ${{\bf{w}}_j} = ({\omega _1},{\omega _2},\ldots,{\omega _n})$ are loadings for the $j$th principal component satisfying ${\bf{w}}_j^\rm{T}{{\bf{w}}_j} = 1$ and ${\bf{w}}_\rm{i}^\rm{T}{\bf{w}}_j = 0$ for $i\neq j$, thus ${z_j} = {\bf{w }}_j^{\rm{T}}{\bf{X}}$ is the $j$th component.

To find out the first principal component, we try to maximize the variance of $z_1$, which is $\rm{var}(z_1)=\rm{var}({\bf{w }}_1^{\rm{T}}{\bf{X}})=\bf{w}_\rm{1}^\rm{T}\rm{var}(\bf{X})\bf{w}_\rm{1}$. We estimate $\rm{var}(\bf{X}\rm{)}$ by the sample co-variance matrix $\bf{S}$, we maximize $L=\bf{w}_\rm{1}^\rm{T}\bf{S}\bf{w}_\rm{1}-\lambda({\bf{w}}_1^\rm{T}{{\bf{w}}_1} - 1)$ where $\lambda$ is the Lagrange multiplier. By taking derivative we arrive at $(\bf{S}-\lambda\bf{I})\bf{w}_\rm{1}=0$. It is obvious $\bf{w}_1$ is an eigenvector of the sample co-variance matrix $\bf{S}$.

Now the problem comes. Solving the equation gets you all eignenvalues and eigenvectors. I searched the internet all materials I found simply tell you to rank the eigenvalues and the eigenvector of the largest eigenvalue is the first principal component, and the eigenvector of the second eigenvalue is the second principal component, and so one so forth.

My question is how do we show or prove the largest eigenvalue corresponds to the largest variance and the second largest eigenvalue corresponds to the second largest variance and so on. Thank you.

  • 3
    $\begingroup$ Hint: The variables will be de-correlated in the transformed space (the new basis are the eigenvectors). $\endgroup$ – Vladislavs Dovgalecs Apr 3 '15 at 7:06