stepwise linear regression on a principal component

Question

What I want to do

I have 500,000 variables for a few hundreds records. I did principal component to find the subgroup of records. From the analysis I find that 4 principal components are sufficient to explain the variance of samples. Now I want to choose a subset of variables (N<100) to represent these principal components. When I have a new sample I can easily and cheaply detect which subgroup it belongs to. The software that does principal component outputs a weight for each varaible for each component to indicate its contribution to the principal component.

What is the problem

These 500,000 are not all independent. So I cannot simply chose the variants with the highest weight.

What is the solution in practice

People suggested to calculate an overall contribution score for each variable. It sums up the weights of this variable weighted by the % of variance explained by each component. Then select the top ranked variables, and remove the correlated ones by a correlation measure r2, say r2>0.8.

What is my proposal

The above method is not idea becaue 1) the main principal component may be over-counted. I redo a principal component analysis with the chosen variable only and see that the first component was replicated, but the second component is completely noise. 2) the r2 cutoff is arbitrary. 3) I don't know how many variable to choose.

Therefore, I proposed to perform a stepwise linear regression. For each component, I do a linear regression on each variable (the component is the dependent variable), then choose the most significant variable to put into the model and do regression again until no more variables are significant after Bonferroni correction. Then I do the second component starting with no variables in the model but adjusting for the first principal component. And then I do the third component adjusting for the first and second component, and so on.

My question

Can I do that?

Answer 1

If I understand you, you want to use stepwise as a method to select the variables that you will collect in the future and then use for future work, right?

This avoids some of the many problems with stepwise regression (discussed here and elsewhere many times) but using statistical significance as a cutoff is a bit problematic for several reasons:

First, any significance level that you use as a cutoff is arbitrary. Why p = 0.05? Or 0.01?

Second, the significance values reported from stepwise are incorrect.

Third, it doesn't seem like significance is the key issue here. As so often is the case, it seems to me that you should be looking at effect size, not significance.

I suggest you include variables until a certain R^2 is reached, or some other criterion. Another alternative is to use LASSO or LAR or some other penalized variable selection method (but this doesn't avoid problems 1 or 3, above).