1
$\begingroup$

I'm running some data exploration and have a large ish number of variables with varying degrees of correlation and covariance. I'd like to start throwing out some variables I don't "need" according to the criteria that most variable and uncorrelated ones are my most important *** (I know this isn't true in general!). I know pca is a poor feature selection choice, and I really only use it for visualization. However, given my criteria, is this a proper application?

It's been awhile since some of my instructional days and I recall a reasonable amount of the linear algebra behind it (using orthogonal decomposition and using the eigenspectrum of a matrix to create a hyper plane whose axes are the directions of maximum variability, the linear scoring used to create the pcs etc etc).

My big question though- Is it necessarily true that high loading scores on a set of factors imply that these factors account for the most covariance in a technical sense (i.e. these are the factors where things are 'most different'). If the former is true- and I want to compare loading across multiple pcs in this sense, where is my breakpoint (i.e suppose I have a low loading on pc 1 for factor A and a high loading for factor B on pc2- if the explained variance due to pc1 is say 70 percent, how can I deduce that more of my data points vary wrt to B than A).

Thanks for all your input. It's been a long day so please let me know where I can clean this up if needed.

$\endgroup$
2
  • $\begingroup$ Sorry I apparently come from a different subfield -- what are "loadings?" And to clarify, you want to keep the variables (not the dimensions in data space) that have the least correlation with (something) $\endgroup$ Commented Oct 5, 2017 at 13:15
  • $\begingroup$ no worries! people sometimes use scores and loadings interchangeably which makes things...confusing to say the least. loadings in this context is each variables contribution to the the principle component as in the context of their linear combination in creating the PC $\endgroup$ Commented Oct 5, 2017 at 16:49

1 Answer 1

1
$\begingroup$
  1. I would say not--PCA is useful if you want a low dimensional projection, but it sounds like what you want is to find a small number of your original features that give you a predictive variable set. For this you want to look into a sparse approach, either at the model level (something with L1 regularization) or at the pre-processing level, by using something like factor analysis.

  2. In a purely mathematical sense this must be true -- if one of the linear combinations has 99% of its magnitude in one original dimension then clearly that dimension accounts for the majority of its projected space. However I don't really see what looking at the loadings give you -- unless your PCA vectors have most of their magnitudes concentrated in a single dimension, you can't say much at all. To look at your example, what you should do -- I would say -- is just compute the variance explained by each of your variables to begin with and don't bother with the PCA.

$\endgroup$
2
  • $\begingroup$ thanks for your reply! on point one, i'm running a cluster algorithm with a looot of variables and I to start my efa I'd like to play with the variables with the highest variance first, and I'd rather not test for the relative equalities of variance between the variables to start from a purely computational stand point (but of a time crunch over here!) $\endgroup$ Commented Oct 9, 2017 at 17:13
  • $\begingroup$ If it's important you use a subset of the variables you currently have, then I'd suggest you simply look at the variance of all of your variables (if that's the criterion of interest) and choose the top x% of them. I'm not really sure what you mean by 'relative equalities of variance' I'm afraid $\endgroup$ Commented Oct 10, 2017 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.