# How to determine the most influential regressor in multiple-linear regression with multi-collinearity?

I have a multivariate regression model with 3 regressors and a response variable. All of them are numeric with same unit. But the 3 regressors are significantly correlated.

I need to find out which regressor contribute most/second to the response variable. I want to keep the regressors intact so I don't want to do something like PCA because each PC would be combination of all 3 regressors.

I tried a package in R called "relaimpo: Relative importance of regressors in linear models". It seems doing it suppose to do by giving back a relative importance matrix.

#

y is genome size while x1,x2 and x3 are components of the genome. So all the 4 variables have same unit. First few lines of my data set is like this:

  y    x1   x2    x3


1 81722 12102 6257 21494

2 79389 12046 5254 21955

3 81467 12759 6012 21811

4 77425 12333 3393 21899

5 78172 12741 5458 21464

My question is: which of the x contribute most to the variability of y?

####if use LM model without interaction, all x significant

summary(lm(y~.,data)) Call: lm(formula = y ~ ., data = data)
Residuals: Min 1Q Median 3Q Max -1921.8 -561.1 79.9 516.0 2720.8 Coefficients:
Estimate Std. Error t value Pr(>|t|)

(Intercept) 5.094e+04 1.731e+03 29.422 < 2e-16 ***

x1 7.924e-01 9.186e-02 8.626 1.27e-09 ***

x2 1.192e+00 2.244e-01 5.313 9.61e-06 ***

## x3 5.847e-01 1.241e-01 4.710 5.28e-05 ***

Residual standard error: 1104 on 30 degrees of freedom
Multiple R-squared: 0.9713, Adjusted R-squared: 0.9684
F-statistic: 338 on 3 and 30 DF, p-value: < 2.2e-16

############################################################

But x1, x2 and x3 are all significantly correlated:

with(data,cor.test(x1,x2))

Pearson's product-moment correlation


data: x1 and x2 t = 6.0258, df = 32, p-value = 1.007e-06 alternative hypothesis: true correlation is not equal to 0 95 percent confidence interval:
0.5188266 0.8561576
sample estimates:
cor
0.7290763

with(data,cor.test(x3,x2))

Pearson's product-moment correlation


data: x3 and x2
t = 10.463, df = 32, p-value = 7.411e-12
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
0.7707769 0.9386146
sample estimates:
cor 0.8796592

with(data,cor.test(x3,x1))

Pearson's product-moment correlation


data: x3 and x1
t = 5.3758, df = 32, p-value = 6.657e-06
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
0.4572082 0.8329872
sample estimates:
cor
0.6888668

if I include interaction in the model, none of x significant

summary(lm(y~x1*x2*x3,data))

Call: lm(formula = y ~ x1 * x2 * x3, data = data)

Residuals:
Min 1Q Median 3Q Max
-1664.4 -670.1 171.6 378.2 1745.1

Coefficients: Estimate Std. Error t value Pr(>|t|)
(Intercept) 4.667e+04 1.682e+04 2.775 0.0101 *
x1 5.139e-01 1.354e+00 0.380 0.7073
x2 1.316e+01 6.909e+00 1.904 0.0680 .
x3 5.592e-01 9.026e-01 0.620 0.5409
x1:x2 -8.294e-04 5.655e-04 -1.467 0.1544
x1:x3 3.576e-05 7.119e-05 0.502 0.6197
x2:x3 -5.360e-04 3.275e-04 -1.637 0.1138
x1:x2:x3 3.583e-08 2.641e-08 1.357 0.1865

Residual standard error: 910.1 on 26 degrees of freedom
Multiple R-squared: 0.9831, Adjusted R-squared: 0.9785
F-statistic: 215.6 on 7 and 26 DF, p-value: < 2.2e-16

######### use "relaimpo" package on model without interaction

rela1=calc.relimp(y~.,data,type = "lmg", rela = TRUE )
rela1 Response variable: y
Total response variance: 38531968
Analysis based on 34 observations

3 Regressors:
x1 x2 x3
Proportion of variance explained by model: 97.13%
Metrics are normalized to sum to 100% (rela=TRUE).

Relative importance metrics:

     lmg


x1 0.3241856
x2 0.3503318
x3 0.3254826

########## use "relaimpo" package on model with interaction

It only accept 2-variable interactions and "lmg" method:

rela2=calc.relimp(y~x1+x2+x3+x1*x2+x1*x3+x2*x3,data,type = "lmg", rela = TRUE ) rela2 Response variable: y
Total response variance: 38531968
Analysis based on 34 observations

6 Regressors: x1 x2 x3 x1:x2 x1:x3 x2:x3
Proportion of variance explained by model: 98.19%
Metrics are normalized to sum to 100% (rela=TRUE).

Relative importance metrics:

          lmg


x1 0.320602721
x2 0.345668344
x3 0.322170263
x1:x2 0.001659694
x1:x3 0.008405496
x2:x3 0.001493482

#

From the relative importance metrics output, it seems x1,x2 and x3 have similar importance for y in this dataset. The result doesn't bother me but I would like to consult experts here whether there is major mistakes in the method.

My questions: Is the multi-collinearity a big problem here? How can I deal with it while keep my xs? Is there other ways to calculate relative importance in the model other than using this R package? Any advice of selecting the interaction terms?

• Can you say a little more about your situation? What are your variables? What do you mean that they have the "same scale"? Can you paste in your regression output? (Also, I cannot parse the sentence, "It seems doing it suppose to do".) – gung Apr 1 '16 at 15:48