I have a set of five variables that are highly correlated (namely SAPS3, SIDA, ALBUM, Lactate), and I must see how they are associated with other variables (I will use a variable named lIL6 as an example).
I have decided to use principal component analysis to deal with collinearity and try to understand how these variables behave.
I decided to use three PC (all with eigenvalues above one). Together, they explain at least 70% of the variance (which is fine for me).
I used the psych package (dbp is the name of the database):
ppp.r<-principal(dbp,nfactors=3,rotate="varimax",scores=T)
This returns the following:
Call: principal(r = dbp, nfactors = 3, rotate = "varimax", scores = T)
Standardized loadings (pattern matrix) based upon correlation matrix
RC1 RC2 RC3 h2 u2
SAPS3 -0.09 0.09 0.98 0.97 0.026
Albumin 0.53 -0.70 -0.12 0.78 0.220
SIDa 0.88 0.04 0.07 0.78 0.217
SIG 0.16 0.92 0.05 0.88 0.119
Lactate -0.65 0.04 0.19 0.46 0.543
RC1 RC2 RC3
SS loadings 1.51 1.35 1.02
Proportion Var 0.30 0.27 0.20
Cumulative Var 0.30 0.57 0.78
Proportion Explained 0.39 0.35 0.26
Cumulative Proportion 0.39 0.74 1.00
I have the following doubts:
1- What is the optimal cutoff to say that a variable has a negligible load on the PC score? I have read 0.50 in some places, while others said 0.60. Can I say, for instance, that the load of SAPS3 on PC1 is negligible? What about the load of Albumin on PC1 (it is 0.53)?
2- Since I want to see how these variables are related to another variable (variable lIL6), I therefore performed a simple linear regression between PC and lIL6. I am particularly interested in PC2 because SIG and Albumin have a high load on it, so I ran this:
lm(formula = lIL6 ~ ppp.r$scores[, 2], data = db)
Residuals:
Min 1Q Median 3Q Max
-1.6547 -0.5058 -0.1231 0.4078 2.3494
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.61663 0.08818 18.332 < 2e-16 ***
ppp.r$scores[, 2] 0.32542 0.08870 3.669 0.000424 ***
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.8225 on 85 degrees of freedom
Multiple R-squared: 0.1367, Adjusted R-squared: 0.1266
F-statistic: 13.46 on 1 and 85 DF, p-value: 0.0004239
So it is significant (I understand that it is not a good model, R-squared is low, etc) Can I state that, since SIG and Albumin have a high load in PC2 and since PC2 is associated with lIL6, SIG and Albumin are probably associated with lIL6 in a manner that is independent of SAPS3 values (considering that SAPS3 has a very low load on PC2)?
lILC6
against the original five variables? Since the third PC still accounts for 20% of all variance, with 100-78 = 22% left to explain, none of the remaining two PCs (not shown) can have more than 20%, whence each has at least 2% (and likely more): this implies the eigenvectors don't vary a whole lot--the ratio of the highest to the lowest is at at most $\sqrt{30\%/2\%} \lt 4$, indicating there should be no multicollinearity problem at all! $\endgroup$