Using PCA to perform feature selection when variability and correlation are the only selection critera?

Question

I'm running some data exploration and have a large ish number of variables with varying degrees of correlation and covariance. I'd like to start throwing out some variables I don't "need" according to the criteria that most variable and uncorrelated ones are my most important *** (I know this isn't true in general!). I know pca is a poor feature selection choice, and I really only use it for visualization. However, given my criteria, is this a proper application?

It's been awhile since some of my instructional days and I recall a reasonable amount of the linear algebra behind it (using orthogonal decomposition and using the eigenspectrum of a matrix to create a hyper plane whose axes are the directions of maximum variability, the linear scoring used to create the pcs etc etc).

My big question though- Is it necessarily true that high loading scores on a set of factors imply that these factors account for the most covariance in a technical sense (i.e. these are the factors where things are 'most different'). If the former is true- and I want to compare loading across multiple pcs in this sense, where is my breakpoint (i.e suppose I have a low loading on pc 1 for factor A and a high loading for factor B on pc2- if the explained variance due to pc1 is say 70 percent, how can I deduce that more of my data points vary wrt to B than A).

Thanks for all your input. It's been a long day so please let me know where I can clean this up if needed.

Answer 1

1. I would say not--PCA is useful if you want a low dimensional projection, but it sounds like what you want is to find a small number of your original features that give you a predictive variable set. For this you want to look into a sparse approach, either at the model level (something with L1 regularization) or at the pre-processing level, by using something like factor analysis.

2. In a purely mathematical sense this must be true -- if one of the linear combinations has 99% of its magnitude in one original dimension then clearly that dimension accounts for the majority of its projected space. However I don't really see what looking at the loadings give you -- unless your PCA vectors have most of their magnitudes concentrated in a single dimension, you can't say much at all. To look at your example, what you should do -- I would say -- is just compute the variance explained by each of your variables to begin with and don't bother with the PCA.