Suppose that we have a SVD for our data matrix centered $X = UDV^T$.

Then it is stated that the i-th principal component, $Xv_i$, has variance $\frac{d_i^2}{N}$. Consider these steps.

$$ var(Xv_i) = var(UDV^Tv_i) \\ = var(d_i u_i) \\ = d_i^2 var(u_i) $$

In order to get that the variance is $\frac{d_i^2}{N}$, we would require $var(u_i) = \frac{1}{N}$. But how is it that this is always true? $u_i$ is a unit vector, so what if we have the unit vector $[1 \ \ 0]^T$, then the variance is 0.25, which is not equivalent to $\frac{1}{2}$?

  • $\begingroup$ What is your source for this assertion? What is (trivially) true is that the mean square of $u_i$ is $1/N,$ so that's likely what the source means by "variance." $\endgroup$ – whuber Jul 17 at 14:56
  • $\begingroup$ @whuber I can't recall the particular source, but I think I've seen it in a couple of places. However, I did just find a statement from Elements of Statistical Learning, 2nd ed., pg 66, Eq. (3.49). It seems to only apply to the first principal component, where $var(z_1) = var(Xv_1) = \frac{d_1^2}{N}$. This appears to suggest that the variance of the normalized first principal component is $var(u_1) = \frac{1}{N}$? And right, I understand that the mean square of $u_i$ would be $\frac{1}{N}$ since it is a unit vector. $\endgroup$ – user5965026 Jul 17 at 16:40
  • $\begingroup$ @whuber However, I can't think of many situations where this would hold for $var(u_1)$. The ones I can think of are the situations where $u_1$ is even sized and there's an equal number of positive and negative $\frac{1}{\sqrt{N}}$ components, in which case the variance would be 1. The mean would be zero, and the sum of squared difference would just be the mean square. $\endgroup$ – user5965026 Jul 17 at 16:43
  • $\begingroup$ For every $i,$ the variance of $u_i$ equals $1/N-|\bar u_i|^2$ where $\bar u_i$ is the mean of the components of $u_i.$ $\endgroup$ – whuber Jul 17 at 17:54
  • 1
    $\begingroup$ @whuber Ah I see. The column space of $X$ is orthogonal to the 1 vector. The columns of $U$ form an orthonormal basis for the column space of $X$, implying that every vector in the basis is always orthogonal to the 1 vector. ESL seems to single out the first principal component as having variance $\frac{d_i^2}{N}$, but like you said, I think this applies to all the eigenvectors $u_i$ with a corresponding non-zero eigenvalue. $\endgroup$ – user5965026 Jul 17 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.