How to use principal components analysis to select variables for regression? - Cross Validated most recent 30 from stats.stackexchange.com 2019-09-16T10:26:15Z https://stats.stackexchange.com/feeds/question/9590 https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/9590 12 How to use principal components analysis to select variables for regression? N26 https://stats.stackexchange.com/users/4054 2011-04-15T17:57:38Z 2015-05-31T07:34:30Z <p>I am currently using principal components analysis to select variables to use in modelling. At the moment, I make measurements A, B and C in my experiments -- What I really want to know is: Can I make fewer measurements and stop recording C and or B to save time and effort? </p> <p>I find that all 3 variables load heavily onto my first principal component which accounts for 60% of the variance in my data. The component scores tell me that if I add these variables together in a certain ratio (aA+bB+cC). I can get a score on PC1 for each case in my dataset and could use this score as a variable in modelling, but that doesn't allow me to stop measuring B and C. </p> <p>If I square the loadings of A and B and C on PC1, I find that variable A accounts for 65% of the variance in PC1 and variable B accounts for 50% of the the variance in PC1 and variable C also 50%, i.e. some of the variance in PC1 accounted for by each variable A, B and C is shared with another variable, but A comes out on top accounting for slightly more. </p> <p>Is it wrong to think that I could just choose variable A or possibly (aA+bB, if necessary) to use in modelling because this variable describes a large proportion of the variance in PC1 and this in turn describes a large proportion of the variance in the data? </p> <p>Which approach have you gone for in the past?</p> <ul> <li>Single variable which loads heaviest on PC1 even if there are other heavy loaders? </li> <li>Component score on PC1 using all variables even if they are all heavy loaders?</li> </ul> https://stats.stackexchange.com/questions/9590/-/9591#9591 14 Answer by whuber for How to use principal components analysis to select variables for regression? whuber https://stats.stackexchange.com/users/919 2011-04-15T19:38:16Z 2011-04-15T20:45:56Z <p>You haven't specified what "modeling" you plan on, but it sounds like you're asking about how to select <em>independent</em> variables among $A$, $B$, and $C$ for the purpose of (say) regressing a fourth <em>dependent</em> variable $W$ on them.</p> <p>To see that this approach <em>can</em> go wrong, consider three independent Normally distributed variables $X$, $Y$, and $Z$ with unit variance. For the <em>true, underlying</em> model choose a small constant $\beta \ll 1$, a really tiny constant $\epsilon \ll \beta$, and let the (dependent variable) $W = Z$ (plus a little bit of error independent of $X$, $Y$, and $Z$).</p> <p>Suppose the independent variables you have are $A = X + \epsilon Y$, $B = X - \epsilon Y$, and $C = \beta Z$. Then $W$ and $C$ are strongly correlated (depending on the variance of the error), because each is close to a multiple of $Z$. However, $W$ is uncorrelated with either of $A$ or $B$. Because $\beta$ is small, the first principal component for $\{A, B, C\}$ is parallel to $X$ with eigenvalue $2 \gg \beta$. $A$ and $B$ load heavily on this component and $C$ loads not at all because it is independent of $X$ (and $Y$). Nevertheless, if you eliminate $C$ from the independent variables, leaving only $A$ and $B$, you will be throwing away <em>all</em> information about the dependent variable because $W$, $A$, and $B$ are independent!</p> <p>This example shows that for regression you want to pay attention to how the independent variables are correlated with the dependent one; you can't get away just by analyzing relationships among the independent variables.</p> https://stats.stackexchange.com/questions/9590/-/9633#9633 4 Answer by Peter Flom for How to use principal components analysis to select variables for regression? Peter Flom https://stats.stackexchange.com/users/686 2011-04-16T22:06:58Z 2011-04-16T22:06:58Z <p>If you have only 3 IVs, why do you want to reduce them?</p> <p>That is, is your sample very small (so that 3 IVs risks overfitting)? In this case, consider partial least squares</p> <p>Or are the measurements very expensive (so, in future, you'd like to measure only one IV)? In this case, I'd consider looking at the different regressions with each IV separately and together.</p> <p>Or did someone in your past over-emphasize the value of parsimony? In this case, why not include all 3 IVs?</p>