2
$\begingroup$

I did a principal component analysis, which resulted in 7 components that I am now using for a principal component regression on my independent variable.

However, I want to add 2 control variables, but these are not component scores, but "normal" variables (on a Likert scale, the same sort of variables my independent variables were).

Is that okay to do or do I have to make these into components as well? I have done the regression both with those two as variables and combined in a component and the results are practically the same.

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ This is not a duplicate. $\endgroup$
    – amoeba
    Commented Jan 18, 2018 at 20:41
  • 2
    $\begingroup$ I have no idea why this is closed as a duplicate of that Q. It is not a duplicate! This is a separate and a clearly defined question (that has been asked before: stats.stackexchange.com/questions/47972 - but wasn't answered, so now that Q is closed as a duplicate of this one). It's well answered below and the answer is accepted. This thread should stay open. I voted to reopen. $\endgroup$
    – amoeba
    Commented Jan 21, 2018 at 21:33

1 Answer 1

Reset to default
1
$\begingroup$

It depends.

As you may know PCR starts with PCA on the independent variables.

$$X = TP' + E$$

After obtaining the scores ($T$), one carries out regression between $T$ and $Y$ so that:

$$Y = TB + F$$

The first PCA step assures the scores (columns of $T$) are uncorrelated so that you can find "healthy" regression coefficients rather than dealing with the originally problematic matrix (rank deficiency, multicollinearity, $p \gg n$ etc..) which can, for example, yield very large regression coefficients and cause overfitting.

Thus, if you add some variables, depending on the nature of those variables, you may end up with a similar problem that caused you to use PCR rather than OLS in the first place. On the other hand, it may be just OK. I suggest to confirm your each model's success via testing it on an independent validation set or at least by using CV.

Personally, I would add those variables prior to PCR. If interpretability by looking at the regression coefficients is your concern, then $$Y = XPB + F$$ thus $\hat{B} = PB$ which you can directly use on your (probably at least mean-centered) data and can be interpreted just as easly.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Well, adding a few (specifically, two) additional variables will not make $p > n$ if PCA originally reduced $p$ to $p\ll n$. Is your concern then that these 2 additional variables might be highly correlated to the retained PC scores? $\endgroup$
    – amoeba
    Commented Jan 18, 2018 at 13:58
  • $\begingroup$ For this specific case, yes. But usually I try to provide less specific answers to aid future readers. Is this a bad practice? $\endgroup$
    – gunakkoc
    Commented Jan 18, 2018 at 14:08
  • $\begingroup$ It's a good practice :-) I was just clarifying what you meant. $\endgroup$
    – amoeba
    Commented Jan 18, 2018 at 14:11
  • $\begingroup$ Thank you, very helpful reply! I have done most of the assumption testing and I think I have good and valid results in the end. $\endgroup$
    – Boudewijn Hulst
    Commented Jan 19, 2018 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.