1
$\begingroup$

I have several dependent variables to analysis. Since the independent variables are highly correlated, I am thinking about using the PCR model. Now I was wondering if by choosing different dependent variables we will get different results? I mean if I use PCR, can I have the same new element that can be considered as a new IV for different dependent variables?

$\endgroup$
3
$\begingroup$

If you are strictly doing principal components (PC) regression, then all you are doing is changing your original set of N correlated predictors into a new set of N PCs, each a linear combination of the original predictors but with each PC orthogonal to all the others. It effectively starts by finding the linear combination of predictors that explains the most variance of the predictors among the cases (first PC), then finding the combination orthogonal to the first PC that explains the most of the remaining variance, and so on. In this case the principal components are simply a function of the original set of predictors and will not change if you use them in models with different dependent variables. This thread is a useful link to further information.

There is, however, a related method called Partial Least Squares (PLS), which incorporates information about the dependent variable to obtain a set of transformed orthogonal predictors. In that case the first component is chosen based on the single-variable relation of each of the predictors to the dependent variable. In PLS the set of transformed orthogonal predictors thus necessarily changes depending on the dependent variable being examined. See Section 6.3 of ISLR for further information.

That said, with correlated predictors you might instead want to consider a different approach, like ridge regression. Principal components are strongly linked to the original data set, and might pose issues if you wish to use PC regression for predictions based on new data. Coefficients in ridge regression are expressed directly in terms of the original predictors, not in terms of a potentially sample-dependent combination of PCs, so they might be easier to apply in prediction. Ridge regression is very closely related to PC regression; instead of simply throwing out the set of PCs least related to outcome and having the same weights for all the retained PCs, you effectively weight the PCs by their relation to the dependent variable, then express coefficients in terms of the original predictors. This relation between PC and ridge regression is described, for example, on page 79 of ESLII.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.