Looking to train a linear regression model on PCA transformed data. While PCA helped reduced the # of predictors variables significantly from 307 to ~40 without compromising much of the variance, I am, however, losing interpretability of the variables. Now my variables are labeled as PC1,PC2...PCn.

The question is two-fold:

  1. How can I use PCA transformed data to train my linear regression model and predict on new test data?

  2. While prediction accuracy is important in this case, I am also interested in variable importance for business purposes.

  • $\begingroup$ Factor analysis (e.g. rotation) would be a better tool for exploratory and variable importance. $\endgroup$ – SmallChess Mar 22 '17 at 4:29

The practice of using the scores of the PCA model (the lower-dimensional variables you can get from PCA) is known as Principal Component Regression (PCR). One shortcoming of PCR is that the transformation matrix that converts the data from its original form to an orthogonal space (the loadings), from which you can select a small number of relevant ones to reduce the dimensonality, is that these loadings are estimated to maximize the variance of the new, orthogonal variables. This ignores the relationship between these variables and the response in your regression problem. Here is a description of how to perform PCR in R.

An alternative is to use Partial Least Squares (PLS). This procedure searches for loadings that give you a dataset of orthogonal variables that are also highly correlated with the response. In a nutshell, you get dimensionality reduction and the latent variables you pick are usually more powerful predictors of your response than the variables from PCR. There is a PLS package for R, though it has limited functionality.

Both PLS and PCA can create problems of interpretation since they transform your original data into new variables. If you are not comfortable interpreting these transformations and have many variables, you could try a variable selection procedure. LASSO is quite popular. One method that I have found useful recently while working with high-dimensional data where I want flexibility and interpretability is MARS, which is implemented in the R package earth. This method will perform variable selection for you, and the results are interpretable almost like in linear regression. However, the models it fits are flexible since they are piecewise linear, which may be too unconstrained for your problem.

  • $\begingroup$ Right - I would like to keep variables that are highly correlated with my response variable as I am looking for interpretability and PCA isn't ideal as it ignores this relationship. $\endgroup$ – keval Mar 22 '17 at 13:42
  • $\begingroup$ Then PLS would be more a more appropriate choice. $\endgroup$ – Deathkill14 Mar 22 '17 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.