(This is a soft question) Recently I'm learning Principal Component Analysis, and it appears to have a lot of issues:
- You have to transform the data to roughly the same scale before applying PCA, but how the feature scaling should be performed is unspecified. Standardization? Scaling to unit length? Log-transformation? Box-Cox transformation? I believe all of them somehow works, but they answer different questions, and it's nontrivial to figure out the transformation given a problem.
- To perform PCA, eigenvalues and eigenvectors must be computed, but the signs of eigenvectors are undetermined. At first sight, SVD could be a good solution, as it gives the same result across different implementations. However, as I understand it, the result of SVD is merely an arbitrary but reproducible choice of eigenvectors.
- Principal components are linear combinations of variables, but do they make sense? I mean, you cannot add a monkey's body temperature to ten times its tail length, because they are of different units. (Speaking of the unit, which unit system should you use is another aspect of my first point)
- When trying to interpret the principal components, should you inspect the loading (coefficient) of the $i$th principal component $y_i$ on the $j$th element $X_j$, or their correlation $\text{corr}(y_i, X_j)$? Rencher (1992) recommends only looking at the coefficients, but as far as I know, there is no consensus on this issue.
To sum up, PCA is a statistical (or arguably mathematical) method that looks quite immature to me, as it introduces numerous subjectivity and bias in throughtout the process. Nonetheless, it remains one of the most widely-used multivariate analysis methods. Why is it? How do people overcome the problems I have raised? Are they even aware of them?
References:
Rencher, A. C. “Interpretation of Canonical Discriminant Functions, Canonical Variates and Principal Components.” The American Statistician, 46 (1992), 217–225.