I'm running some data exploration and have a large ish number of variables with varying degrees of correlation and covariance. I'd like to start throwing out some variables I don't "need" according to the criteria that most variable and uncorrelated ones are my most important *** (I know this isn't true in general!). I know pca is a poor feature selection choice, and I really only use it for visualization. However, given my criteria, is this a proper application?
It's been awhile since some of my instructional days and I recall a reasonable amount of the linear algebra behind it (using orthogonal decomposition and using the eigenspectrum of a matrix to create a hyper plane whose axes are the directions of maximum variability, the linear scoring used to create the pcs etc etc).
My big question though- Is it necessarily true that high loading scores on a set of factors imply that these factors account for the most covariance in a technical sense (i.e. these are the factors where things are 'most different'). If the former is true- and I want to compare loading across multiple pcs in this sense, where is my breakpoint (i.e suppose I have a low loading on pc 1 for factor A and a high loading for factor B on pc2- if the explained variance due to pc1 is say 70 percent, how can I deduce that more of my data points vary wrt to B than A).
Thanks for all your input. It's been a long day so please let me know where I can clean this up if needed.