28
$\begingroup$

Suppose $u$ is the vector that maximizes the variance of the projection of the data with design matrix $X$.

Now, I have seen materials that refer $u$ as the (first) principal component of the data, which is also the eigenvector with largest eigenvalue.

However, I have also seen that the principal component of the data is $X u$.

Obviously, $u$ and and $Xu$ are different things. Can anyone help me out here and tell me what is the difference between these two definitions of principal components?

$\endgroup$
2
  • $\begingroup$ Eigenvector u is the direction of the axis (values of u are the direction cosines relative the original axes). Xu is the data itself, the values of the principal component, the coordinates onto the aforementioned axis). $\endgroup$
    – ttnphns
    Feb 27, 2014 at 15:37
  • $\begingroup$ What are the dimensions of $X$? There exist different conventions for the design matrix: one is $n_\mathrm{sample} \times n_\mathrm{features}$, another is is $n_\mathrm{features} \times n_\mathrm{sample}$. $\endgroup$ Jan 24, 2023 at 19:11

1 Answer 1

36
$\begingroup$

You are absolutely correct in observing that even though $\mathbf{u}$ (one of the eigenvectors of the covariance matrix, e.g. the first one) and $\mathbf{X}\mathbf{u}$ (projection of the data onto the 1-dimensional subspace spanned by $\mathbf{u}$) are two different things, both of them are often called "principal component", sometimes even in the same text.

In most cases it is clear from the context what exactly is meant. In some rare cases, however, it can indeed be quite confusing, e.g. when some related techniques (such as sparse PCA or CCA) are discussed, where different directions $\mathbf{u}_i$ do not have to be orthogonal. In this case a statement like "components are orthogonal" has very different meanings depending on whether it refers to axes or projections.

I would advocate calling $\mathbf{u}$ a "principal axis" or a "principal direction", and $\mathbf{X}\mathbf{u}$ a "principal component".

I have also seen $\mathbf u$ called "principal component vector".

I should mention that the alternative convention is to call $\mathbf u$ "principal component" and $\mathbf{Xu}$ "principal component scores".

Summary of the two conventions:

$$\begin{array}{c|c|c} & \text{Convention 1} & \text{Convention 2} \\ \hline \mathbf u & \begin{cases}\text{principal axis}\\ \text{principal direction}\\ \text{principal component vector}\end{cases} & \text{principal component} \\ \mathbf{Xu} & \text{principal component} & \text{principal component scores} \end{array}$$


Note: Only eigenvectors of the covariance matrix corresponding to non-zero eigenvalues can be called principal directions/components. If the covariance matrix is low rank, it will have one or more zero eigenvalues; corresponding eigenvectors (and corresponding projections that are constant zero) should not be called principal directions/components. See some discussion in my answer here.

$\endgroup$
6
  • 3
    $\begingroup$ Convention 2 should be outlawed. It has the capacity to create no end of confusion for beginners as it conflates basis vectors and components of data vectors with respect to the basis. $\endgroup$ May 9, 2017 at 11:53
  • 1
    $\begingroup$ what about the Loadings definition ? Are loadings the individual values of the eigenvector u ? $\endgroup$
    – seralouk
    Mar 12, 2018 at 16:23
  • $\begingroup$ @sera See stats.stackexchange.com/questions/143905 and stats.stackexchange.com/questions/125684 $\endgroup$
    – amoeba
    Mar 12, 2018 at 16:29
  • $\begingroup$ @amoeba thank you! one last question. In SVD, for X = USVh (Vh: V transposed) if the eigenvectors are the columns of U, then can I call Vh as loadings? $\endgroup$
    – seralouk
    Mar 12, 2018 at 17:11
  • $\begingroup$ @sera No. See stats.stackexchange.com/questions/134282 $\endgroup$
    – amoeba
    Mar 12, 2018 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.