What I want to do
I have 500,000 variables for a few hundreds records. I did principal component to find the subgroup of records. From the analysis I find that 4 principal components are sufficient to explain the variance of samples. Now I want to choose a subset of variables (N<100) to represent these principal components. When I have a new sample I can easily and cheaply detect which subgroup it belongs to. The software that does principal component outputs a weight for each varaible for each component to indicate its contribution to the principal component.
What is the problem
These 500,000 are not all independent. So I cannot simply chose the variants with the highest weight.
What is the solution in practice
People suggested to calculate an overall contribution score for each variable. It sums up the weights of this variable weighted by the % of variance explained by each component. Then select the top ranked variables, and remove the correlated ones by a correlation measure r2, say r2>0.8.
What is my proposal
The above method is not idea becaue 1) the main principal component may be over-counted. I redo a principal component analysis with the chosen variable only and see that the first component was replicated, but the second component is completely noise. 2) the r2 cutoff is arbitrary. 3) I don't know how many variable to choose.
Therefore, I proposed to perform a stepwise linear regression. For each component, I do a linear regression on each variable (the component is the dependent variable), then choose the most significant variable to put into the model and do regression again until no more variables are significant after Bonferroni correction. Then I do the second component starting with no variables in the model but adjusting for the first principal component. And then I do the third component adjusting for the first and second component, and so on.
Can I do that?