I was reading this Reddit thread on principal component analysis (PCA). One user responded with the following:


There is nothing that would suggest that PC's are meaningful. Quite the opposite, because they are forced to be orthogonal to each other you pretty much guarantee that they will NOT be meaningful at all.

Unless the phenomenon you're modeling also happens to have orthogonal features, such as axis of rotation so pitch, yaw, roll that are orthogonal to each other, but this pretty much never happens.

Trying to interpret principal components is like the classical newbie mistake. People do it because some ancient statistical software implemented PCA as a special case of factor analysis and they confuse FA with PCA and mix/match them and really mean FA when they say PCA in their papers.

Is it true that PCA assumes that your features are orthogonal? Does this mean that PCA is not a good technique when features are not orthogonal?

EDIT: Given the (now deleted) comment, I want to say that I have no idea whether this is referring to the original features or the latent features. Furthermore, since I'm a novice to this, I don't even have a good idea of what the difference between these would be. If your answer addresses this, I would appreciate an explanation of the differences between original and latent features in this context.


So I believe it's wrong, and I am not sure if you misunderstood what it's claiming. Ie the claim is that the features extracted are orthogonal and therefore unlikely to be relevant...( There is no guarantee that the features your model requires are orthogonal)

PCA is used as a dimensionality reduction method. It finds the direction that maximises correlation, then finds the next direction orthogonal to the previous directions maximising correlation etc.and repeat.

If you are are using a linear method or neural network ( with inner product) the orthogonality of the resulting features doesn't matter, since the model can create any linear combination anyway.

The point though, of PCA, is that many signal and image processing tasks can be characterized as a Signal, correlated across inputs(pixels) and independent noise.

Selecting the directions of maximal correlation therefore is filtering for the signal and throwing away noise.

In terms of inputs, you should use PCA, precisely when your data is correlated and non orthogonal.

I would agree with the comment that the actual directions are unlikely to be meaningful, but I would say that is irrelevant: what's important is the sub space covered by the directions. in terms of the Reddit thread where it mentioned finding clusters in the 3d set of pcas, it's the subspace that matters not the specifics of the axes chosen

  • $\begingroup$ Thanks for the answer. There are a number of things that are unclear to me. 1. When you say the "features extracted", are you referring to the original features or latent features? 2. And when you say "In terms of inputs, you should use PCA", what are you referring to as "inputs" here? 3. Also, can you please elaborate on why PCA should not be used on the outputs (and, related to 2., can you please clarify what "outputs" would mean in this case)? $\endgroup$ – The Pointer Jan 30 at 15:50
  • 1
    $\begingroup$ 1. The principal components, latent features. 2. I am saying you should use PCA when your original data is correlated. The post on Reddit was saying don't try to interpret the latent features, because they are orthogonal. $\endgroup$ – seanv507 Jan 30 at 16:01
  • $\begingroup$ Ok, I see what you mean. Thanks. $\endgroup$ – The Pointer Jan 30 at 16:02
  • 1
    $\begingroup$ Outputs: typically the output is a single number so you would not use pca $\endgroup$ – seanv507 Jan 30 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.