# How to select significant variables for multi-factor regression?

I have a set of data $Y \in \mathbb{R}^n$ and a set of $k$ factors $\{F_1, ..., F_K\}$ with $F_i \in \mathbb{R}^n ~ \forall i$.

I would like to perform a factor analysis which consits in finding $\beta \in \mathbb{R}^k$ such that:

$$Y_t = \alpha + \sum_{i=1}^k \beta_i {F_i}_t + \varepsilon_t$$

I believe that I would need to make sure that the factors are independent from each other, and I wonder what process (statistical test) I should use to determine which factors I shall remove from the framework because they are too correlated.

There are two main questions:

1. How do I find and remove the factors that are not linearly independent?
2. How do I determine which of the remaining factors are significant?
-
I think you have to conduct model selection to check which variables-factors can be removed. For this purpose, you have an ocean of possibilities. –  user10525 May 23 '12 at 13:47
Probably worth plotting factor pairwise to check dependence visually - not all important dependencies are linear. R's pairs() works well for this. –  naught101 May 27 '12 at 8:58

Variable selection usually has a criteria for inclusion that relates to goodness of fit or prediction. To a different question on variable selection I gave detailed references to books on the topic. I also mentioned new work by Lacey Gunter on variable selection using a criteria for qualitative interactions (applied primarily in medical research for individualized teratment). Her research and some joint work I did with her are summarized in a forthcoming monograph on variable selection to be published by Springer later this year (hopefully). But it could be that multicollinearity is reason here for reducing factors which might not fit with the variable selection literature. If there is a single factor that is almost a linear combination of n others you can probalbly throw out any one of the n+1 factors. Which one you choose doesn't matter. If you have two separate sets then you can remove one for eahc of the tow sets and so on.

-
Can we have the link of the answers you mention? –  SRKX May 24 '12 at 6:49
I have somee of this in my answer to a recent question "Applying an Interaction term to all the IVs". But the other book references were in a discussion on another site that has statistical discussions. So here is a list of books we (Lacey and I) referenced in our forthcoming monograph on variable selection:Burnham, K. P., and Anderson, D. R. (1998). Model Selection and Inference: A Practical Information-Theoretic Approach. Springer-Verlag, New York.Lahiri, P. editor (2001). Model Selection. Institute of Mathematical Statistics Lecture Notes, 38, Beachwood, Ohio. –  Michael Chernick May 24 '12 at 16:11
Linhart, H. and Zucchini, W. (1986). Model Selection. Wiley, New York. Miller, A. J. (1990). Subset Selection in Regression. Chapman & Hall, London. –  Michael Chernick May 24 '12 at 16:13
Thanks for the links, do you have a test you would recommend to find which factors are to be grouped together? Or is a simple look at the correlation matrix enough? –  SRKX May 27 '12 at 9:36
I don't recall the specifics but there are diagnostics that check for collinearity. See the Wiley text Regression Diagnostics (1980/ by Balsley, Kuh and Welsch. –  Michael Chernick May 27 '12 at 12:44