# Interpreting principal component loads for linear regression in R

I have a set of five variables that are highly correlated (namely SAPS3, SIDA, ALBUM, Lactate), and I must see how they are associated with other variables (I will use a variable named lIL6 as an example).

I have decided to use principal component analysis to deal with collinearity and try to understand how these variables behave.
I decided to use three PC (all with eigenvalues above one). Together, they explain at least 70% of the variance (which is fine for me).
I used the psych package (dbp is the name of the database):

ppp.r<-principal(dbp,nfactors=3,rotate="varimax",scores=T)


This returns the following:

Call: principal(r = dbp, nfactors = 3, rotate = "varimax", scores = T)
Standardized loadings (pattern matrix) based upon correlation matrix

RC1   RC2   RC3   h2    u2
SAPS3   -0.09  0.09  0.98 0.97 0.026
Albumin  0.53 -0.70 -0.12 0.78 0.220
SIDa     0.88  0.04  0.07 0.78 0.217
SIG      0.16  0.92  0.05 0.88 0.119
Lactate -0.65  0.04  0.19 0.46 0.543

RC1  RC2  RC3
SS loadings           1.51 1.35 1.02
Proportion Var        0.30 0.27 0.20
Cumulative Var        0.30 0.57 0.78
Proportion Explained  0.39 0.35 0.26
Cumulative Proportion 0.39 0.74 1.00


I have the following doubts:

1- What is the optimal cutoff to say that a variable has a negligible load on the PC score? I have read 0.50 in some places, while others said 0.60. Can I say, for instance, that the load of SAPS3 on PC1 is negligible? What about the load of Albumin on PC1 (it is 0.53)?

2- Since I want to see how these variables are related to another variable (variable lIL6), I therefore performed a simple linear regression between PC and lIL6. I am particularly interested in PC2 because SIG and Albumin have a high load on it, so I ran this:

lm(formula = lIL6 ~ ppp.r$scores[, 2], data = db) Residuals: Min 1Q Median 3Q Max -1.6547 -0.5058 -0.1231 0.4078 2.3494 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 1.61663 0.08818 18.332 < 2e-16 *** ppp.r$scores[, 2]  0.32542    0.08870   3.669 0.000424 ***

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.8225 on 85 degrees of freedom
Multiple R-squared: 0.1367,    Adjusted R-squared: 0.1266
F-statistic: 13.46 on 1 and 85 DF,  p-value: 0.0004239


So it is significant (I understand that it is not a good model, R-squared is low, etc) Can I state that, since SIG and Albumin have a high load in PC2 and since PC2 is associated with lIL6, SIG and Albumin are probably associated with lIL6 in a manner that is independent of SAPS3 values (considering that SAPS3 has a very low load on PC2)?

• What problem(s) do you run into when you simply regress lILC6 against the original five variables? Since the third PC still accounts for 20% of all variance, with 100-78 = 22% left to explain, none of the remaining two PCs (not shown) can have more than 20%, whence each has at least 2% (and likely more): this implies the eigenvectors don't vary a whole lot--the ratio of the highest to the lowest is at at most $\sqrt{30\%/2\%} \lt 4$, indicating there should be no multicollinearity problem at all! – whuber Nov 18 '13 at 21:40

## 1 Answer

Answered partially in comments:

What problem(s) do you run into when you simply regress lILC6 against the original five variables? Since the third PC still accounts for 20% of all variance, with 100-78 = 22% left to explain, none of the remaining two PCs (not shown) can have more than 20%, whence each has at least 2% (and likely more): this implies the eigenvectors don't vary a whole lot--the ratio of the highest to the lowest is at at most $\sqrt{30\%/2\%} \lt 4$, indicating there should be no multicollinearity problem at all! – whuber