Let's say you have a data set with GPA (dependent variable) and Amount of alcohol, Amount of study, IQ, and SAT score as the independent variables. And you want to perform the principal component analysis in R for dimension reduction.

In the csv file you read in R, do you have to have GPA or do you have to remove it?


If your goal is dimensionality reduction, then you should not include GPA in PCA. Dimensionality reduction typically refers to reducing the dimension of your feature space or your input space. We do not include our dependent variable in this, only the independent variables.

If you included GPA in PCA and then dropped one or more of your principal components, then you would be dropping some information from GPA as well. This is likely not what you intend to do, so it would be inappropriate to include GPA when you do PCA.

  • $\begingroup$ thank you so much for your reply. What I have is a data set of the dependent variable (Y) with about 20 independent variables (Xs). My ultimate goal is to choose the best Xs to explain Y (I can also form nonlinear combinations or linear combination/manipulation of the Xs too). As you can see, I have many (= 20) Xs and hope that I can perform some dimension reduction. $\endgroup$ – Jun Jang Feb 15 '18 at 3:11
  • $\begingroup$ towardsdatascience.com/… You might find this helpful. :) $\endgroup$ – Matt Brems Feb 15 '18 at 3:11
  • 1
    $\begingroup$ @JunJang 20 predictors isn't that many unless you have fewer than 200 or so cases. If your interest is in prediction then you probably should include as many predictors as are reasonably related to GPA so that you don't end up with omitted-variable bias. $\endgroup$ – EdM Feb 15 '18 at 3:23
  • $\begingroup$ @EdM I have 66 rows of data. Would you recommend PCA? Some of the variables are definitely highly correlated (I used Excel to create the correlation matrix and I see some numbers close to 0.90). By the way, the actual data I have are not GPA data. I just used them for simplicity. The actual data I have are financial data. $\endgroup$ – Jun Jang Feb 15 '18 at 3:25
  • 1
    $\begingroup$ @JunJang for prediction you would probably be better off using ridge regression. It's related to PCA but instead of completely removing some prinpipal components it weights them according to their relation to outcome, and regression coefficients are directly expressed in terms of the original predictors rather than the hard-to-interpret principal components. $\endgroup$ – EdM Feb 15 '18 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.