When doing principal components regression, do I need to standardize independent variables and/or dependent variable?

Question

I want to run PCA on a set of variables and then regress my dependent variable on the PCA scores.

I have the following questions:

Should I scale and center my variables?
If yes, should I also standardize my dependent variable before running linear regression?
What if I don't standardize my dependent variable?

regress the scores on my dependent variable Not clear enough so far. Are you regressing the "dependent variable" on the components? Or regressing a chosen component on that variable? — ttnphns, Mar 31, 2014 at 11:47
@ttnphns after I run PCA, I will get principal components and the corresponding scores. Now, I want to regress these principal component scores on my dependent variable. — darkage, Mar 31, 2014 at 11:49
If you are not shrinking or penalising the regression in "PCA space" this will achieve nothing. PCA is an "affine" transformation (I think) of your X matrix, and it is reversible/invertible. Its like modelling $ Y=A^T (X-c)\gamma + e $ (where $A$ and $ c $ are functions of $ X $) instead of $ Y=X\beta +e $. — probabilityislogic, Mar 31, 2014 at 11:50
You'll usually just feed the (small number of) components to the regression that account for, say, 90% of the variation in the original data, as given by the cumulative sum of squares of the eigenvalues. I assume this is what the OP had in mind. — Stephan Kolassa, Mar 31, 2014 at 11:52

Stephan Kolassa · Accepted Answer · 2014-03-31 11:58:19Z

3

Yes. If you don't scale and center, your PCA will not capture covariances, but the directions of largest variation. Put differently: your PCA should really not depend on whether lengths are measured in meters or inches, or your temperatures in Celsius and Fahrenheit.
PCA works on the independent data only, it doesn't care about the dependent variables. You will usually not transform the dependent variables. Or you may want to transform them independently of the PCA, e.g., to stabilize variances or take logs, depending on the underlying science.
See 2. You will usually not transform the dependent variable just because you do a PCA on the independent variables. So the question of whether or not to transform the DV and whether or not Bad Things will happen if you transform (or don't transform) is orthogonal to the PCA.

answered Mar 31, 2014 at 11:58

Stephan Kolassa

124k20 gold badges248 silver badges474 bronze badges

1

$\begingroup$ 1.(a) Sometimes when the variables are measured on comparable scales the directions of largest variance are thought to be more interesting. $\endgroup$
– Scortchi - Reinstate Monica ♦
Mar 31, 2014 at 14:14
$\begingroup$ @Scortchi and @Stephan Kolassa : Now after I regress the scores of the first PC on the dependent variable, I get a model Y = intercept + a*PC1(note that PCA was performed on standardised X matrix) what will be the interpretation of the model? and how will I interpret the fitted values obtained? $\endgroup$
– darkage
Apr 1, 2014 at 6:11
1

$\begingroup$ The interpretation of the model will depend on the interpretation (if any) of your principal components. Examine the loadings and see whether you can interpret the PCs in a meaningful way. PCA is notoriously hard to interpret. You may want to consult your favorite textbook. The fitted values have the same interpretation as always: they are fitted values. $\endgroup$
– Stephan Kolassa
Apr 1, 2014 at 9:15

Add a comment |

1 Answer 1