2
$\begingroup$

Running some form of regression on an input dataset that exhibits strong multicollinearity can cause unstable regression coefficients, because the regression algorithm can somewhat arbitrarily attribute importance to each predictor in a set of collinear predictors.

Principal Component Regression (PCR) is often used to 'solve' this problem by describing the input dataset according to a set of orthogonal axes - the principal components - which thus do not exhibit multicollinearity.

From what I have read, the PCR procedure is to eliminate the principal components that account for only small variations in the input dataset, then run OLS regression on the principal components. This 'solves' multicollinearity because the coefficients are stable due to the orthogonality of the predictor axes.

However, I have read that doing PCR on all principal components is equivalent to doing OLS on the original data. This suggests that the issue of multicollinearity is therefore not dealt with this 'complete' PCR. I don't understand why? All of the principal components are orthogonal and hence no multicollinearity is present. Indeed, with 'complete' PCR, all of the input data has been included (i.e. no information loss) and therefore using all components should only improve things.

Of course, one of the primary reasons for doing 'incomplete' PCA/PCR is dimensionality reduction, but I want to consider only the issue of multicollinearity. Am I missing something key here?

$\endgroup$
1
  • $\begingroup$ Re: "... and hence no multicollinearity is present." That is incorrect. Consider the extreme case where the model matrix is entirely zeros. PCA will produce an (arbitrary) orthonormal frame of principal components from such a matrix -- but that doesn't change the collinearity in your original design matrix one whit. The point is that the eigenvectors with zero (or essentially zero, to numerical precision) eigenvalues represent dimensions not included in your explanatory variables. $\endgroup$
    – whuber
    Dec 24, 2023 at 13:39

1 Answer 1

2
$\begingroup$

I don't know if it's necessary to remove some PCs, but, with colinear data, the later PCs are probably not going to be useful. Here is an example in R:

install.packages("pls")
library(pls)
library(olsrr)
set.seed(1234)

x1 <- rnorm(1000)   
x2 <- x1 + rnorm(1000, 0, .1)
x3 <- x2 + rnorm(1000, 0, .1)
x4 <- x3 + rnorm(1000, 0, .1)
y <- x1 + x2 + x3 + x4 + rnorm(1000)

m1 <- lm(y~x1 + x2 + x3 + x4)
ols_eigen_cindex(m1)  #Shows moderately high condition index of 36

pcr_model <- pcr(y~x1 + x2 + x3 + x4, scale = TRUE, validation = "CV") #PC regression
summary(pcr_model) 

This yields:

VALIDATION: RMSEP
Cross-validated using 10 random segments.
       (Intercept)  1 comps  2 comps  3 comps  4 comps
CV           4.111   0.9734   0.9729   0.9736   0.9741
adjCV        4.111   0.9733   0.9728   0.9734   0.9739

TRAINING: % variance explained
   1 comps  2 comps  3 comps  4 comps
X    99.41    99.81    99.93   100.00
y    94.41    94.42    94.42    94.43

showing that the 2nd, 3rd, and 4th components add almost nothing.

EDIT: Note that this last line of output is about the relationships of the components with the dependent variable (nice touch by the programmers!). The first PC accounts for 94.41% of the variance in y; the remaining three add only 0.02%.

In addition, those PCs are likely to be very hard to interpret. They are getting what is "left over" after the first PC, but the first PC got almost everything.

The amount of information lost is very minimal.

$\endgroup$
5
  • $\begingroup$ This is potentially misleading because your use of "amount of information" is not directly relevant to PCR, which concerns finding linear combinations of explanatory variables that are not orthogonal to the response variable. You are using "amount of information" outside that context in which the response variable is not considered at all. This is a popular misconception about PCR. $\endgroup$
    – whuber
    Dec 24, 2023 at 13:41
  • $\begingroup$ @whuber Well, I usually suggest PLS instead of PCR for that precise reason, but, in this case, the first PC was pretty much everything. Even though it didn't account for relationships with the DV, it's hard to see how the later components could add anything at all. $\endgroup$
    – Peter Flom
    Dec 24, 2023 at 14:04
  • $\begingroup$ Re "hard to see:" stats.stackexchange.com/a/9591/919. $\endgroup$
    – whuber
    Dec 24, 2023 at 14:32
  • $\begingroup$ I did some checking and the pcr output includes the relation between the PCs and the DV, it is given in the line marked "Y". The first component accounts for 94.41 % of the variation in Y. The next three add 0.02%. $\endgroup$
    – Peter Flom
    Dec 24, 2023 at 15:33
  • 1
    $\begingroup$ Good! It would be helpful in your post to explain how the output supports your interpretation, which would make this a very good answer. $\endgroup$
    – whuber
    Dec 24, 2023 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.