1
$\begingroup$

First off, apologies if this is basic but I'm still learning about PCA. I am somewhat confused as to what exactly PCA can provide that I can't find out through other means. For example:

  1. Dimensionality Reduction/Redundancy: couldn't one just look at the covariance matrix? If we wanted to retain our original feature set but wanted to reduce it in some way, why not just pull out extraneous variables that are highly correlated?
  2. Visualization at lower dimensions/clustering: why not just run a clustering algorithm and look at in-cluster and between-cluster statistics?

The only thing I can think of that may be more useful would be if one wanted to use the latent representations or principal components for actual use somehow, like in regression. But even then I'm not sure how common this is, and I never really read about this.

$\endgroup$
2
  • 1
    $\begingroup$ pca is "just look[ing] at the covariance matrix"; it just allows us to see correlations between more than two of variables at a time :) $\endgroup$ Nov 5, 2022 at 17:26
  • 1
    $\begingroup$ Suppose your data are three columns: one of 1s, one a 50/50 mix of 1s and 0s and one that is 1 minus the second column. Together, these three columns are linearly dependent, but the correlation of each pair is low. // Your PCA-for-regression idea is called partial-least-squares, although I suppose you could naively use PCA first and regression second, for some reason. $\endgroup$
    – Sycorax
    Nov 5, 2022 at 17:35

1 Answer 1

0
$\begingroup$

"If we wanted to retain our original feature set but wanted to reduce it in some way, why not just pull out extraneous variables that are highly correlated?" PCA gives you linear combinations that represent an as high as possible amount of variation in the lower dimension. "Pulling out" some of your variables won't be as good in terms of representing the variation, and also PCA informs you what the maximum is that can be achieved.

"Visualization at lower dimensions/clustering: why not just run a clustering algorithm and look at in-cluster and between-cluster statistics?" Looking at statistics does not give you a visualisation, and there are a lot of thing you cannot see from this, such as potential nonlinear shapes, outliers etc. PCA is a linear projection method, so the resulting image can be fairly intuitively interpreted (although of course we have to be careful because of the information loss, which is inevitable looking at higher dimensions in a low dimensional plot).

By the way, whenever I do clustering, I look at visualisations (potentially, but not only and not necessarily always PCA) to visually validate my clustering in the first place. There are many ways a clustering can go wrong, and visual validation can reveal at least some of them.

$\endgroup$
2
  • $\begingroup$ Thanks. It seems most explanations for "reducing dimensionality in the dataset" seem to imply that we can learn which features are worth removing, ie. feature reduction. What you mention is "representing the variation", which I get, but I'm still not entirely sure how that is more beneficial towards feature reduction per say. $\endgroup$
    – vctrm
    Nov 7, 2022 at 4:49
  • $\begingroup$ @vctrm A major decision when doing dimension reduction is whether you want to remove and keep some of the original features, or whether you're fine with linear combinations of the original features. PCA delivers linear combinations. This is a problem in some applications, as it means that you still need to observe all original features, and may be harder to interpret. But in some other applications summarising your original features into linear combinations may be just fine. $\endgroup$ Nov 7, 2022 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.