0
$\begingroup$

I'm sorry if this has been asked before. I searched the archives but couldn't find a similar question although it looks like a simple one.

I am fitting a mixed effect regression model. Consider that this is my full model:

m1 = lmer(y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8+ x9 + x10 + (1 | Subject), data = df)

And the output of this full model is as follows. Note that I used lmerTest packacage for the calculation of p values:

              Estimate Std. Error         df t value Pr(>|t|)   
(Intercept) -1.712e-02  1.433e-01  1.920e+03  -0.119  0.90491   
x1          -2.376e-02  8.999e-03  7.910e+02  -2.640  0.00845 **
x2           2.524e-02  1.243e-02  3.544e+02   2.031  0.04297 * 
x3           5.638e-03  8.853e-03  2.182e+03   0.637  0.52428   
x4           9.676e-03  1.387e-02  2.182e+03   0.698  0.48553   
x5           9.014e-03  6.298e-03  2.082e+03   1.431  0.15255   
x6          -3.017e-02  1.717e-02  2.199e+03  -1.757  0.07906  
x7          -5.920e-03  5.524e-03  2.189e+03  -1.072  0.28397   
x8           1.082e-02  9.891e-03  2.191e+03   1.094  0.27396   
x9           2.736e-02  1.680e-02  2.052e+03   1.629  0.10342   
x10          1.239e-02  1.257e-02  2.174e+03   0.986  0.32432

Therefore, both x1 and x2 seem to be the significant effects. But then I dropped x1 and x2 respectively to build two reduced models.

m2 = lmer(y ~ x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + (1 | Subject), data = df)

m3 = lmer(y ~ x1 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + (1 | Subject), data = df)

I compared these reduced models against the full model using ANOVA. Only x1 improved the model, not x2.

anova(m1, m2)

Models:
..1: y ~ x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + (1 | Subject)
object: y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + (1 | Subject)
       Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)   
..1    12 901.57 970.16 -438.79   877.57                            
object 13 896.06 970.37 -435.03   870.06 7.5068      1   0.006146 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


anova(m1, m3)

Models:
..1: y ~ x1 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + (1 | Subject)
object: y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + (1 | Subject)
       Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)
..1    11 896.54 959.42 -437.27   874.54                         
object 13 896.06 970.37 -435.03   870.06 4.4747      2     0.1067

This is the output of the first reduced model, that is m2, without x1:

Fixed effects:
              Estimate Std. Error         df t value Pr(>|t|)  
(Intercept) -6.403e-02  1.424e-01  2.228e+03  -0.450   0.6529  
x2           1.313e-02  8.458e-03  2.210e+03   1.552   0.1207  
x3           1.488e-02  1.374e-02  2.209e+03   1.083   0.2789  
x4           1.147e-02  6.187e-03  2.215e+03   1.854   0.0639 
x5          -1.920e-02  1.673e-02  2.206e+03  -1.148   0.2512  
x6          -4.886e-03  5.544e-03  2.237e+03  -0.881   0.3782  
x7          -6.745e-03  7.405e-03  2.215e+03  -0.911   0.3625  
x8           2.394e-02  1.668e-02  2.210e+03   1.435   0.1515  
x9           1.243e-02  1.260e-02  2.221e+03   0.987   0.3237  
x10          1.917e-02  1.196e-02  2.207e+03   1.602   0.1092

And this is the output of the second reduced model, that is m3, without x2:

Fixed effects:
              Estimate Std. Error         df t value Pr(>|t|)   
(Intercept)  9.813e-02  9.574e-02  2.241e+03   1.025  0.30549   
x1          -2.156e-02  8.098e-03  2.215e+03  -2.662  0.00781 **
x3           1.072e-02  1.385e-02  2.209e+03   0.773  0.43936   
x4           6.864e-03  6.218e-03  2.215e+03   1.104  0.26980   
x5          -2.797e-02  1.696e-02  2.208e+03  -1.649  0.09937  
x6          -6.422e-03  5.526e-03  2.238e+03  -1.162  0.24529   
x7           6.368e-03  9.610e-03  2.218e+03   0.663  0.50761   
x8           3.044e-02  1.527e-02  2.206e+03   1.994  0.04632 * 
x9           7.137e-03  1.218e-02  2.223e+03   0.586  0.55813 

How is it possible? Is it because x2 has a relatively low t (and therefore, high p) value? And under these circumstances, is x2 a significant predictor or not?

Thank you.

Update: Upon the request of @Tahir, I added ANOVA results and outputs of two reduced models.

$\endgroup$
4
  • $\begingroup$ Could you provide outputs of other two reduced models and anova results? $\endgroup$
    – T.E.G.
    Dec 5, 2016 at 20:37
  • $\begingroup$ Thank you very much @Tahir. I provided them. Actually, the original models had more predictor variables but I had removed them for the sake of clarity. I added these variables as well. Now these are the original models I used in analysis. $\endgroup$
    – pulser
    Dec 6, 2016 at 12:03
  • $\begingroup$ How correlated are x1 and x2? $\endgroup$
    – jwimberley
    Dec 6, 2016 at 12:32
  • $\begingroup$ r = 0.19, p < .05. In fact, I had checked collinearity based on VIF and dropped variables with a VIF higher than 4. x1 has a VIF of 3.16 and x2 has a VIF 1.97. $\endgroup$
    – pulser
    Dec 6, 2016 at 12:51

1 Answer 1

1
$\begingroup$

Note that x2 is only "significant" in a model that includes the significant x1. As this thread shows, adding one significant predictor to a model can let a formerly insignificant predictor become significant even if the two predictors are not collinear. In your example, x1 and x2 are correlated, so this type of behavior is particularly likely. It doesn't take much correlation among predictors for this to occur; you shouldn't put too much trust in measures like VIF. A high VIF may be a warning sign, but a low VIF does not necessarily protect you from such seemingly counterintuitive results.

Follow links from the thread noted above for many other examples of correlated predictors coming in and out of "significance" depending on the presence of others.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.