Use of PCA analysis to selection of variables for a regression analysis

Question

I have too many environmental variables to use in a multiple regression analysis. If I use all the variables to models are just to complex. The use of the PCA axis in the regression analysis was impossible to interpret (since there wasn't a clear correlation to environmental variables), so we chose to select a limited number of variables, namely the ones that had the higher explanation in PCA... A PCA was used for each set of environmental variables, namely the variables related with the structure of the stream, the evolving vegetation, the climatic, the physical-chemical characteristics of the water from summer period and from winter period, separately. The PCA was performed using the correlation matrix option, using the software PC-ORD, v. 4.21 (McCune & Mefford 1999). For each set of variables, only the variables with coordinates higher to 0.20, for the two first axis, of the CA, were selected to be used in Multiple regression analysis. I could not found literature that confirmes that that's ok to do this, but I think it is not wrong. Can anyone help me?

Best regards, Leena

Answer 1

When it comes to PCA, relatively few approaches are objectively wrong. But I see two potential problems with your approach. First, suppose a variable has no high loading on any obtained component. That means it is basically independent from the other variables in the analysis. Does this mean it will not contribute to your regression? Not necessarily. Suppose it fills "its own niche" very well in accounting for some of the variance in your outcome. After all, that outcome was not included in the PCA, right?

The use of the PCA axis in the regression analysis was impossible to interpret (since there wasn't a clear correlation to environmental variables), so we chose to select a limited number of variables, namely the ones that had the higher explanation in PCA.

The last part of this seems contradictory to me. If you want to use individual variables as predictors rather than the poorly-performing components that summarize dimensions in the data, the most consistent thing to do is to disregard the PCA results when selecting predictors. You care about how a predictor relates to the outcome, not about ensuring that different predictors are correlated with each other.

Another note: it sounds as if you have conducted or will conduct multiple iterations of regression as you attempt to find good predictors. You'll want to read, on this site or elsewhere, discussions of ways to avoid the pitfalls that come with that. I'd recommend looking at posts on this site with the tag "variable selection."

Answer 2

As @Rolando2 said, the whole area of variable selection comes up here a lot.

But one method that gets to the substantive problem he has with PCA is partial least squares; if you are using SAS, there is PROC PLS.

Answer 3

The most recent article I have seen on this problem is

• Cumming, J.A. and Wooff, D.A. (2007)Dimension reduction via principal variables, Computational Statistics & Data Analysis, 52, 1:550-565.

A good earlier review was provided in chapter 6 of

• Jolliffe, IT(2002). Principal Component Analysis. Springer-Verlag, New York, 2nd edition.