2
$\begingroup$

I have too many environmental variables to use in a multiple regression analysis. If I use all the variables the models are just too complex. The use of the PCA axes in the regression analysis was impossible to interpret (since there wasn't a clear correlation with environmental variables), so we chose to select a limited number of variables, namely those that had the higher explanation in PCA.

A PCA was used for each set of environmental variables, namely the variables related with the structure of the stream, the evolving vegetation, the climatic, the physical-chemical characteristics of the water from summer period and from winter period, separately. The PCA was performed using the correlation matrix option, using the software PC-ORD, v. 4.21 (McCune & Mefford 1999). For each set of variables, only the variables with coordinates higher than 0.20, for the two first axes, of the PCA, were selected to be used in multiple regression analysis.

I could not find literature that confirms that it's OK to do this, but I think it is not wrong.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ welcome to the website. What is your analysis goal? I can read two options between the lines. (1) You want to select the variables that contribute most into explaining the outcome. (2) You want to have all variables in the model, as you know that all of them matter, but you are concerned about multicollinearity and large standard errors that result. For either of these tasks, yours is suboptimal procedure (and frankly the rest of the world uses R, rather than some 12-year old package). $\endgroup$
    – StasK
    Feb 29, 2012 at 16:29
  • $\begingroup$ The 2012 study reported at r4stats.com/articles/popularity showed 31% of data analysts using R. $\endgroup$
    – rolando2
    Mar 11, 2013 at 23:29

3 Answers 3

Reset to default
8
$\begingroup$

When it comes to PCA, relatively few approaches are objectively wrong. But I see two potential problems with your approach. First, suppose a variable has no high loading on any obtained component. That means it is basically independent from the other variables in the analysis. Does this mean it will not contribute to your regression? Not necessarily. Suppose it fills "its own niche" very well in accounting for some of the variance in your outcome. After all, that outcome was not included in the PCA, right?

The use of the PCA axes in the regression analysis was impossible to interpret (since there wasn't a clear correlation with environmental variables), so we chose to select a limited number of variables, namely those that had the higher explanation in PCA.

The last part of this seems contradictory to me. If you want to use individual variables as predictors rather than the poorly-performing components that summarize dimensions in the data, the most consistent thing to do is to disregard the PCA results when selecting predictors. You care about how a predictor relates to the outcome, not about ensuring that different predictors are correlated with each other.

Another note: it sounds as if you have conducted or will conduct multiple iterations of regression as you attempt to find good predictors. You'll want to read, on this site or elsewhere, discussions of ways to avoid the pitfalls that come with that. I'd recommend looking at posts on this site with the tag "variable selection."

Improve this answer
$\endgroup$
2
$\begingroup$

The most recent article I have seen on this problem is

  • Cumming, J.A. and Wooff, D.A. (2007) Dimension reduction via principal variables. Computational Statistics & Data Analysis 52: 550-565.

A good earlier review was provided in chapter 6 of

  • Jolliffe, I.T. (2002). Principal Component Analysis. Springer-Verlag, New York, 2nd edition.
Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Thank you for those references. Would you mind summarizing what they say? $\endgroup$
    – whuber ♦
    Mar 11, 2013 at 20:34
  • 1
    $\begingroup$ Sorry for the delay. I just saw your request. <p> The Cumming Article provides a scheme for selectively choosing variables that explain the maximum remaining variance. The first variable is the one that explains the most. Then you would condition the remaining vars on that, etc. then repeat. <b> The Joliffe chapter reviews an whole series of methods, which I don't recall in any detail. I've been happy with the Cumming/Wolfe approach. $\endgroup$
    – Bill Raynor
    Jul 10, 2014 at 20:16
1
$\begingroup$

As @Rolando2 said, the whole area of variable selection comes up here a lot.

But one method that gets to the substantive problem he has with PCA is partial least squares; if you are using SAS, there is PROC PLS.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.