2
$\begingroup$

Let's say you have a data set with GPA (dependent variable) and Amount of alcohol, Amount of study, IQ, and SAT score as the independent variables. And you want to perform the principal component analysis in R for dimension reduction.

In the csv file you read in R, do you have to have GPA or do you have to remove it?

$\endgroup$

1 Answer 1

3
$\begingroup$

If your goal is dimensionality reduction, then you should not include GPA in PCA. Dimensionality reduction typically refers to reducing the dimension of your feature space or your input space. We do not include our dependent variable in this, only the independent variables.

If you included GPA in PCA and then dropped one or more of your principal components, then you would be dropping some information from GPA as well. This is likely not what you intend to do, so it would be inappropriate to include GPA when you do PCA.

$\endgroup$
5
  • $\begingroup$ thank you so much for your reply. What I have is a data set of the dependent variable (Y) with about 20 independent variables (Xs). My ultimate goal is to choose the best Xs to explain Y (I can also form nonlinear combinations or linear combination/manipulation of the Xs too). As you can see, I have many (= 20) Xs and hope that I can perform some dimension reduction. $\endgroup$
    – JungleDiff
    Commented Feb 15, 2018 at 3:11
  • $\begingroup$ towardsdatascience.com/… You might find this helpful. :) $\endgroup$
    – Matt Brems
    Commented Feb 15, 2018 at 3:11
  • 1
    $\begingroup$ @JunJang 20 predictors isn't that many unless you have fewer than 200 or so cases. If your interest is in prediction then you probably should include as many predictors as are reasonably related to GPA so that you don't end up with omitted-variable bias. $\endgroup$
    – EdM
    Commented Feb 15, 2018 at 3:23
  • $\begingroup$ @EdM I have 66 rows of data. Would you recommend PCA? Some of the variables are definitely highly correlated (I used Excel to create the correlation matrix and I see some numbers close to 0.90). By the way, the actual data I have are not GPA data. I just used them for simplicity. The actual data I have are financial data. $\endgroup$
    – JungleDiff
    Commented Feb 15, 2018 at 3:25
  • 1
    $\begingroup$ @JunJang for prediction you would probably be better off using ridge regression. It's related to PCA but instead of completely removing some prinpipal components it weights them according to their relation to outcome, and regression coefficients are directly expressed in terms of the original predictors rather than the hard-to-interpret principal components. $\endgroup$
    – EdM
    Commented Feb 15, 2018 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.