One of the data sets I am working upon had 3 variables which were having almost 100% correlation among themselves. Since I am learning regression modelling I thought I'll do principal component regression. However there is another variable in my data set whose partial residual plot suggested that I use a cubic term for it in my model. The t-test p-value is significant for it at 95% level.

So now I am wondering if it makes sense for me to include polynomial terms of predictors while doing PCA? because I want to assign meaning to principal components, I am not sure how will I assign meaning to a principal component which is made of polynomial terms of original variables. Or should I first calculate Principal Components and then if needed include polynomial terms of principal components in regression?

Apologies if this isn't the right kind of post for this forum.

  • 4
    $\begingroup$ It is a right kind of question for this form, but a kind of question that is rather difficult to answer. In principle you can use anything you want as predictors in your regression: raw variables, principal components, principal components of a subset of variables and cubic terms of other variables -- anything goes (as long as you don't overfit your dataset). How you can assign meaning to principal components, we cannot say unless you tell us what all your predictors (and your DV) are. $\endgroup$
    – amoeba
    Dec 28, 2014 at 14:13
  • $\begingroup$ @Amoeba. Right. I think the main question is whether it can be useful to add a variable in both linear and quadratic form in PCA before regression. Which seems to be an interesting question. $\endgroup$
    – Michael M
    Dec 28, 2014 at 17:39
  • $\begingroup$ @Michael, as the question is currently stated, it says that there are 3 inter-correlated predictors (that OP wants to reduce via PCA) and an additional (4th) predictor that OP wants to transform with a cubic. I don't see a problem in using PCA on the first three and add the 4th one raised to the third power. I also don't see a problem in doing PCA on all four, and I don't understand how it can be more difficult to interpret PCA when using 3rd power of a predictor than when using 1st power of it. What you are asking is about running PCA on both 1st and 3rd power... $\endgroup$
    – amoeba
    Dec 28, 2014 at 18:02
  • $\begingroup$ @amoeba thanks. I just realized that I can always calculate the regression coefficients for original variables from the the coefficients of principal components. So it will be a good idea for me to add polynomial terms of mean centered variables, do PCR and calculate original coefficients back. I am actually going to standardize them too because my variables have very different scales and I don't want to my PC scores to be just made of the variable with most variance. $\endgroup$
    – Durin
    Dec 28, 2014 at 18:02
  • $\begingroup$ But how many predictors do you have? If it's only four, and 3 of them are nearly 100% correlated (as it sounds from your question), then why not simply average those three and that's it? $\endgroup$
    – amoeba
    Dec 29, 2014 at 0:22


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.