I often see the principal components of PCA described as "linear combinations of the original features".
Say we want to compute the principal components of our $m \times n$ design matrix $A$ ($m$ instances, $n$ features), with $m>n$ and rank$(A)=n-1$. Let's assume it's mean-centered for simplicity. We will get an orthogonal $n\times n$ matrix $V$ whose columns are the principal components.
How come we obtained $n$ independent vectors as a result of linearly combining a set of $n-1$ independent vectors? We can't, of course. That last vector was simply one that was constrained to be orthogonal to the rest, not a linear combination of our features (it's in fact in the kernel of the subspace they generate!).
So, why use the interpretation mentioned at the beginning of the post if:
- It is a wrong statement when our features are not linearly independent, at least if we interpret our design matrix as the result from randomly combining our original features (our matrix would thus contain the coefficients of our data with respect to the canonical basis of an $n$-dimensional space).
- It does not seem to be helpful in interpreting the output. In the case where rank$(A)=n$, then the principal components are indeed linear combinations of our features... as is any set of vectors in that vector space, which makes the statement rather sterile, in my opinion. Why not interpret them as a basis of orthonormal vectors of the space spanned by our features?