A problem with interpretation of principal components as linear combinations of features

Question

I often see the principal components of PCA described as "linear combinations of the original features".

Say we want to compute the principal components of our $m \times n$ design matrix $A$ ($m$ instances, $n$ features), with $m>n$ and rank$(A)=n-1$. Let's assume it's mean-centered for simplicity. We will get an orthogonal $n\times n$ matrix $V$ whose columns are the principal components.

How come we obtained $n$ independent vectors as a result of linearly combining a set of $n-1$ independent vectors? We can't, of course. That last vector was simply one that was constrained to be orthogonal to the rest, not a linear combination of our features (it's in fact in the kernel of the subspace they generate!).

So, why use the interpretation mentioned at the beginning of the post if:

1. It is a wrong statement when our features are not linearly independent, at least if we interpret our design matrix as the result from randomly combining our original features (our matrix would thus contain the coefficients of our data with respect to the canonical basis of an $n$-dimensional space).
2. It does not seem to be helpful in interpreting the output. In the case where rank$(A)=n$, then the principal components are indeed linear combinations of our features... as is any set of vectors in that vector space, which makes the statement rather sterile, in my opinion. Why not interpret them as a basis of orthonormal vectors of the space spanned by our features?

Answer 1

I think it is correct to say that "principal components are linear combinations of the original features".

You consider a $m\times n$ data matrix $A$ with $m$ data points in the $n$-dimensional space that has rank $r<n$. And you say that even though the original features are only $r$-dimensional, PCA will extract $n$ principal components, hence the quoted statement cannot be correct.

The problem in your argument is that in this case PCA will not extract $n$ principal components; it will only extract $r$ of them.

The $n\times n$ covariance matrix $C$ will be of rank $r$, meaning that it will have $n-r$ zero eigenvalues. If we do its eigenvector decomposition $C=VSV^\top$, the diagonal matrix $S$ will have these zeros in it. I would only call "principal components" those eigenvectors (columns of $V$) that correspond to non-zero eigenvalues. The ones that correspond to zero eigenvalues do not deserve to be -- and are not -- called principal components.

This resolves the contradiction.

Note that this is not my personal interpretation, it is the standard terminology usage; see e.g. Why are there only $n-1$ principal components for $n$ data points if the number of dimensions is larger or equal than $n$?

Edit in response to the comment: Regarding what exactly is called "principal component" please see my answer here What exactly is called "principal component" in PCA?. Whatever your personal preference is, there are only $r$ principal components if the rank of the data matrix is $r$.