I often see the principal components of PCA described as "linear combinations of the original features".

Say we want to compute the principal components of our $m \times n$ design matrix $A$ ($m$ instances, $n$ features), with $m>n$ and rank$(A)=n-1$. Let's assume it's mean-centered for simplicity. We will get an orthogonal $n\times n$ matrix $V$ whose columns are the principal components.

How come we obtained $n$ independent vectors as a result of linearly combining a set of $n-1$ independent vectors? We can't, of course. That last vector was simply one that was constrained to be orthogonal to the rest, not a linear combination of our features (it's in fact in the kernel of the subspace they generate!).

So, why use the interpretation mentioned at the beginning of the post if:

  1. It is a wrong statement when our features are not linearly independent, at least if we interpret our design matrix as the result from randomly combining our original features (our matrix would thus contain the coefficients of our data with respect to the canonical basis of an $n$-dimensional space).
  2. It does not seem to be helpful in interpreting the output. In the case where rank$(A)=n$, then the principal components are indeed linear combinations of our features... as is any set of vectors in that vector space, which makes the statement rather sterile, in my opinion. Why not interpret them as a basis of orthonormal vectors of the space spanned by our features?
  • $\begingroup$ $m$ instances, $n$ features. Edited. $\endgroup$ – broncoAbierto Jul 21 '16 at 10:28
  • $\begingroup$ I assume that's the rank, and yes, let's assume it's centered. $\endgroup$ – broncoAbierto Jul 21 '16 at 10:29
  • $\begingroup$ I assume that's the rank because it helps me make my point, by the way, although it is not a requisite. $\endgroup$ – broncoAbierto Jul 21 '16 at 10:33
  • $\begingroup$ I'm thinking that part of the confusion might stem from the way I interpret features. If we interpret them as random variables, then the new random variables resulting from expressing the data in the orthogonal basis (i.e. the scores) are indeed linear combinations of the original ones. If we interpret features as a set of vectors which form a basis that we randomly combine to form an observations, then the principal components (i.e. the alternative basis) are not all linear combinations of the original vectors. $\endgroup$ – broncoAbierto Jul 21 '16 at 10:51

I think it is correct to say that "principal components are linear combinations of the original features".

You consider a $m\times n$ data matrix $A$ with $m$ data points in the $n$-dimensional space that has rank $r<n$. And you say that even though the original features are only $r$-dimensional, PCA will extract $n$ principal components, hence the quoted statement cannot be correct.

The problem in your argument is that in this case PCA will not extract $n$ principal components; it will only extract $r$ of them.

The $n\times n$ covariance matrix $C$ will be of rank $r$, meaning that it will have $n-r$ zero eigenvalues. If we do its eigenvector decomposition $C=VSV^\top$, the diagonal matrix $S$ will have these zeros in it. I would only call "principal components" those eigenvectors (columns of $V$) that correspond to non-zero eigenvalues. The ones that correspond to zero eigenvalues do not deserve to be -- and are not -- called principal components.

This resolves the contradiction.

Note that this is not my personal interpretation, it is the standard terminology usage; see e.g. Why are there only $n-1$ principal components for $n$ data points if the number of dimensions is larger or equal than $n$?

Edit in response to the comment: Regarding what exactly is called "principal component" please see my answer here What exactly is called "principal component" in PCA?. Whatever your personal preference is, there are only $r$ principal components if the rank of the data matrix is $r$.

  • $\begingroup$ Actually, after revisiting Jolliffe, Ian. Principal component analysis. John Wiley & Sons, Ltd, 2002. the matter is clear. It is there stated that "The $k$-th derived variable, $\alpha^T x$ is the $k$-th PC." Therefore, the principal components are the new random variables, not the vectors of the orthogonal basis. What confused me is that in R, which I was using to test my assumptions, the columns of the transformation matrix are called PCs. Therefore, I assumed that PCs were commonly defined as these vectors. $\endgroup$ – broncoAbierto Jul 22 '16 at 8:14
  • 1
    $\begingroup$ @broncoAbierto This is just a terminology issue, I have a whole answer about that: stats.stackexchange.com/a/88137/28666 - you might want to take a look. Some people call eigenvectors ($\boldsymbol \alpha$ vector) "principal components", some people call projections of the data onto these eigenvectors "principal components". Nothing of this changes the fact that if the rank of the data matrix is $r$, then there are $r$ principal components and not more. $\endgroup$ – amoeba Jul 22 '16 at 11:30
  • $\begingroup$ that in R, which I was using to test my assumptions, the columns of the transformation matrix are called PCs. I concur with amoeba's statement. "PCs" are (constructed or latent) variables or real dimensions (axes) in the data space. Columns of eigenvector matrix correspond to PCs but are not PCs "thermselves". The columns which correspond to zero, nonexistent PCs are filled with computational dross which should be normally ignored and not called "PCs". Just cut those columns out. [Btw R's terminology often deserves a sour smile.] $\endgroup$ – ttnphns Jul 22 '16 at 11:49
  • $\begingroup$ Yes, it seems that it was simply a terminology issue. Thanks for pointing to that answer. There is something I'd like to point out, though. If by PCs we are referring to eigenvectors, then there will certainly be $n$ (not $r$) such components. Some of those will just be eigenvectors associated with eigenvalue zero and in this context they are of little use, but they do indeed exist. $\endgroup$ – broncoAbierto Jul 22 '16 at 13:07
  • 1
    $\begingroup$ @amoeba Yes, definitely. If we are talking about PC's then there's no point in considering zero-valued eigenvectors. $\endgroup$ – broncoAbierto Jul 22 '16 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.