9
$\begingroup$

(This is a soft question) Recently I'm learning Principal Component Analysis, and it appears to have a lot of issues:

  1. You have to transform the data to roughly the same scale before applying PCA, but how the feature scaling should be performed is unspecified. Standardization? Scaling to unit length? Log-transformation? Box-Cox transformation? I believe all of them somehow works, but they answer different questions, and it's nontrivial to figure out the transformation given a problem.
  2. To perform PCA, eigenvalues and eigenvectors must be computed, but the signs of eigenvectors are undetermined. At first sight, SVD could be a good solution, as it gives the same result across different implementations. However, as I understand it, the result of SVD is merely an arbitrary but reproducible choice of eigenvectors.
  3. Principal components are linear combinations of variables, but do they make sense? I mean, you cannot add a monkey's body temperature to ten times its tail length, because they are of different units. (Speaking of the unit, which unit system should you use is another aspect of my first point)
  4. When trying to interpret the principal components, should you inspect the loading (coefficient) of the $i$th principal component $y_i$ on the $j$th element $X_j$, or their correlation $\text{corr}(y_i, X_j)$? Rencher (1992) recommends only looking at the coefficients, but as far as I know, there is no consensus on this issue.

To sum up, PCA is a statistical (or arguably mathematical) method that looks quite immature to me, as it introduces numerous subjectivity and bias in throughtout the process. Nonetheless, it remains one of the most widely-used multivariate analysis methods. Why is it? How do people overcome the problems I have raised? Are they even aware of them?

References:

Rencher, A. C. “Interpretation of Canonical Discriminant Functions, Canonical Variates and Principal Components.” The American Statistician, 46 (1992), 217–225.

$\endgroup$
10
  • $\begingroup$ PCA is quite mature, but your issue # 1 is extremely important. You can deal OK with the other issues, e.g., using a simple linear regression to restate the PCs in terms of raw variables. There are also several ways to approximate PCs for descriptive/decoding purposes. I go into some of these in my Regression Modeling Strategies book and course notes. $\endgroup$ May 13, 2019 at 2:16
  • 2
    $\begingroup$ Issue 2: Why do people use square roots when they have so many issues? If you take square root of 4, it can be 2, but it can also be -2. At first sight, taking positive value could be a good solution, but it is merely an arbitrary but reproducible choice of sign. Square roots look quite immature to me. $\endgroup$
    – amoeba
    May 13, 2019 at 6:52
  • 1
    $\begingroup$ @amoeba In the context of PCA, issue #2 can be much more serious IMO. If you only use the first principal component, then as in the case of square root, there are 2 possible outcomes (+, -). However, if you consider $p$ principal components, you'll have $p$ undetermined signs, resulting in $2^p$ different results. For $p = 3$, there are +++, ++-, +-+, +--, .-++, -+-, --+, ---, which is already a lot! $\endgroup$
    – nalzok
    May 13, 2019 at 6:58
  • 5
    $\begingroup$ "Sign-arbitrariness" is merely an artifact of how we represent the PCA results. There is no arbitrariness to the PCA itself: the eigenspaces it works with are perfectly well defined. Issues (1) and (3) are advantages of PCA, because they allow one to use subject-matter knowledge and the objectives of the analysis appropriately. Referring to this as "immature" rather misses the entire point of statistical analysis, IMHO, which is to solve real problems in creative and principled ways (as opposed to dumping data into black boxes). $\endgroup$
    – whuber
    May 13, 2019 at 15:59
  • 4
    $\begingroup$ What I don't see here mentioned yet is that many use PCA the same way you'd use a histogram, density plot, or scatter plot: A means to quickly inspect data, rather than a final solution to a problem. PCA is useful for this purpose as the number of dimensions grows, but of course is more informative if care is taken in choosing whether and how to scale. $\endgroup$ May 14, 2019 at 2:49

1 Answer 1

5
$\begingroup$

"Sign-arbitrariness" is merely an artifact of how we represent the PCA results. There is no arbitrariness to the PCA itself: the eigenspaces it works with are perfectly well defined. Issues (1) and (3) are advantages of PCA, because they allow one to use subject-matter knowledge and the objectives of the analysis appropriately. Referring to this as "immature" rather misses the entire point of statistical analysis, IMHO, which is to solve real problems in creative and principled ways (as opposed to dumping data into black boxes).

– whuber

What I don't see here mentioned yet is that many use PCA the same way you'd use a histogram, density plot, or scatter plot: A means to quickly inspect data, rather than a final solution to a problem. PCA is useful for this purpose as the number of dimensions grows, but of course is more informative if care is taken in choosing whether and how to scale.

– Frans Rodenburg

$\endgroup$
3
  • 2
    $\begingroup$ I've copied these comments as a community wiki answer because they are, more or less, answers to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions. $\endgroup$
    – mkt
    Jun 26, 2019 at 12:06
  • 3
    $\begingroup$ +1. You don't have to make such answers CW, btw; but it's of course fine if you prefer to do so. $\endgroup$
    – amoeba
    Jun 26, 2019 at 12:51
  • $\begingroup$ @amoeba Thanks, I'd generally feel more comfortable doing that if I added some original contribution. But I'll keep it in mind. $\endgroup$
    – mkt
    Jun 26, 2019 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.