- That a variable has high variance does not imply that it's objectively important!
- That a variable has low variance does not imply that it's objectively unimportant!
Trivial example: hair color and blood loss in an emergency room
Imagine we have data collected from an emergency room:
- $x_1$ measures hair color.
- $x_2$ measures blood loss.
Just for hypothetical purposes, let's say the covariance matrix is:
$$\Sigma = \left[ \begin{array}{cc} 100 & 0 \\ 0 & 1\end{array} \right] $$
- There's tons of variation in hair color $x_1$.
- On the other hand, blood loss is quite rare. Expected squared deviations from mean blood loss (i.e. the variance) is quite low.
You can instantly see where this is going. An eigenvalue decomposition is:
$$ \left[ \begin{array} {cc} 1 & 0 \\ 0 & 1 \end{array} \right] \left[ \begin{array} {cc} 100 & 0 \\ 0 & 1 \end{array} \right] \left[ \begin{array} {cc} 1 & 0 \\ 0 & 1 \end{array} \right] $$
The first principal component will be $\left[ \begin{array}{c} 1 \\ 0 \end{array} \right]$, the eigenvector associated with the largest eigenvalue.. Hair color describes most of the variation in our data, but that's almost certainly irrelevant in this context!
Disclaimer/note: I have no medical training, and undoubtedly the medical specifics of this hypothetical are all screwed up. Don't take that part literally. The point is about Principal Component Analysis (PCA).