1
$\begingroup$

I am searching for correlations between a dependent variable and a factor or a combination of factors in a repeated measure design. So I used lme() function in R. However, I am getting very different results depending on whether I add on various factors to the the lme formula as compared to when only one is present. If a factor is found to be significant, shouldn't it remain significant when more factors are introduced in the model?

I give an example of the outputs I get using the two models. In the first model I use one single factor:

library(nlme)
summary(lme(Mode ~ Weight, data = Gravel_ds, random = ~1 | Subject))
Linear mixed-effects model fit by REML
 Data: Gravel_ds 
      AIC      BIC   logLik
  2119.28 2130.154 -1055.64

Random effects:
 Formula: ~1 | Subject
        (Intercept) Residual
StdDev:    1952.495 2496.424

Fixed effects: Mode ~ Weight 
                Value Std.Error DF   t-value p-value
(Intercept) 10308.966 2319.0711 95  4.445299   0.000
Weight        -99.036   32.3094 17 -3.065233   0.007
 Correlation: 
       (Intr)
Weight -0.976

Standardized Within-Group Residuals:
        Min          Q1         Med          Q3         Max 
-1.74326719 -0.41379593 -0.06508451  0.39578734  2.27406649 

Number of Observations: 114
Number of Groups: 19 

As you can see, the p-value for factor Weight is significant. This is the second model, in which I add various factors for searching their correlations:

library(nlme)
summary(lme(Mode ~ Weight*Height*Shoe_Size*BMI, data = Gravel_ds, random = ~1 | Subject))
Linear mixed-effects model fit by REML
 Data: Gravel_ds 
       AIC      BIC    logLik
  1975.165 2021.694 -969.5825

Random effects:
 Formula: ~1 | Subject
        (Intercept) Residual
StdDev:    1.127993 2494.826

Fixed effects: Mode ~ Weight * Height * Shoe_Size * BMI 
                                Value Std.Error DF    t-value p-value
(Intercept)                   5115955  10546313 95  0.4850941  0.6287
Weight                      -13651237   6939242  3 -1.9672518  0.1438
Height                         -18678     53202  3 -0.3510740  0.7487
Shoe_Size                       93427    213737  3  0.4371115  0.6916
BMI                         -13011088   7148969  3 -1.8199949  0.1663
Weight:Height                   28128     14191  3  1.9820883  0.1418
Weight:Shoe_Size               351453    186304  3  1.8864467  0.1557
Height:Shoe_Size                 -783      1073  3 -0.7298797  0.5183
Weight:BMI                      19475     11425  3  1.7045450  0.1868
Height:BMI                     226512    118364  3  1.9136867  0.1516
Shoe_Size:BMI                  329377    190294  3  1.7308827  0.1819
Weight:Height:Shoe_Size          -706       371  3 -1.9014817  0.1534
Weight:Height:BMI                -109        63  3 -1.7258742  0.1828
Weight:Shoe_Size:BMI             -273       201  3 -1.3596421  0.2671
Height:Shoe_Size:BMI            -5858      3200  3 -1.8306771  0.1646
Weight:Height:Shoe_Size:BMI         2         1  3  1.3891782  0.2589
 Correlation: 
                            (Intr) Weight Height Sho_Sz BMI    Wght:H Wg:S_S Hg:S_S Wg:BMI Hg:BMI S_S:BM Wg:H:S_S W:H:BM W:S_S: H:S_S:
Weight                      -0.895                                                                                                    
Height                      -0.996  0.869                                                                                             
Shoe_Size                   -0.930  0.694  0.933                                                                                      
BMI                         -0.911  0.998  0.887  0.720                                                                               
Weight:Height                0.894 -1.000 -0.867 -0.692 -0.997                                                                        
Weight:Shoe_Size             0.898 -0.997 -0.873 -0.700 -0.999  0.995                                                                 
Height:Shoe_Size             0.890 -0.612 -0.904 -0.991 -0.641  0.609  0.619                                                          
Weight:BMI                   0.911 -0.976 -0.887 -0.715 -0.972  0.980  0.965  0.637                                                   
Height:BMI                   0.900 -1.000 -0.875 -0.703 -0.999  0.999  0.999  0.622  0.973                                            
Shoe_Size:BMI                0.912 -0.992 -0.889 -0.726 -0.997  0.988  0.998  0.649  0.958  0.995                                     
Weight:Height:Shoe_Size     -0.901  0.999  0.876  0.704  1.000 -0.997 -1.000 -0.623 -0.971 -1.000 -0.997                              
Weight:Height:BMI           -0.908  0.978  0.886  0.704  0.974 -0.982 -0.968 -0.627 -0.999 -0.975 -0.961  0.973                       
Weight:Shoe_Size:BMI        -0.949  0.941  0.928  0.818  0.940 -0.946 -0.927 -0.751 -0.980 -0.938 -0.924  0.935    0.974              
Height:Shoe_Size:BMI        -0.901  0.995  0.878  0.707  0.998 -0.992 -1.000 -0.627 -0.960 -0.997 -0.999  0.999    0.964  0.923       
Weight:Height:Shoe_Size:BMI  0.952 -0.948 -0.933 -0.812 -0.947  0.953  0.935  0.747  0.985  0.946  0.932 -0.943   -0.980 -0.999 -0.931

Standardized Within-Group Residuals:
        Min          Q1         Med          Q3         Max 
-2.03523736 -0.47889716 -0.02149143  0.41118126  2.20012158 

Number of Observations: 114
Number of Groups: 19 

This time the p-value associated to Weight is not significant anymore. Why? Which analysis should I trust?

In addition, while in the first output the field "value" (which should give me the slope) is -99.036 in the second output it is -13651237. Why are they so different? The one in the first output is the one that seems definitively more reasonable to me.

$\endgroup$
2
$\begingroup$

It's not in general true that "if a factor is found to be significant, [it will] remain significant also when more factors are introduced in the model".

There are quite a few possible reasons for this that are relevant to your problem.

First, it may be helpful to remember the distinction between factors and parameters. For example, Weight is just one factor but is modeled with 1 parameter in model 1 and 8 parameters in model 2. Consequently if you want to see (or test) the effect of Weight in the second model you'll need to explicitly compute the relevant marginal effect, by setting the things it is interacted with to particular values.

In model 1 the relevant marginal effect is given by the parameter value, but that's only because the model is linear and additive. You were expecting that effects would always line up one-to-one with parameters this way, but in models more complicated that your first, they just won't.

Second, your second model has a lot more parameters but just the same amount of data, so it's possible that you don't have enough data to reliably estimate everything that you could estimate if you were to assume a simpler model of the data generating process, such as your first model.

Third, you have variables that are quite tightly linearly related to one another (keyword: collinearity). If you put them all in a model then it may easily be that you can only estimate the effects of combinations of them, e.g. heavy tall people vs light small people. However, the tests that you are looking at the results of are parameter by parameter and do not reflect these combinations. Your significance levels may therefore be all over the place. Those levels are then correct answers to a question you are not actually interested in. In this case you just need more different information to disentangle things.

Finally, if some factors exert their effects through their effects on other factors, then a significant effect of a causally distant effect will be removed when a more causally proximate one is added to the model.

It's worth noting that none of these are a consequence of using mixed effects or nlme. They're just normal issues with regression models.

$\endgroup$
  • $\begingroup$ Thanks a lot, your answer is really informative! However, it is not clear to me how should I proceed in my analysis then. I guess have enough data since I have 19 subjects and 6 repetitions for the investigated stimulus for a total of 114 measurements. I want to know whether participants’ choices of the dependent variable depend on their weight, height, shoe size and the combination of those effects. Would the formula of model 2 be correct? Which results should I take into account then? Those of model 2? What is your recommendation? $\endgroup$ – L_T Jul 21 '15 at 0:18
  • $\begingroup$ I think I understood the marginality principle and now my functions are: lme_centroid <- lme(Mode ~ Weight*Height, data = Gravel_ds, random = ~1 | Subject) followed by anova.lme(lme_centroid,type = "marginal"). However, while I get significance for this model for some factors, I do not get significance if I add a third factor in the formula. I do not know how to proceed. You were mentioning that additional information is required in order to disentangle the effects. Can you be more specific please? $\endgroup$ – L_T Jul 21 '15 at 13:34
  • $\begingroup$ Is stats.stackexchange.com/questions/31027/… helpful? $\endgroup$ – conjugateprior Jul 23 '15 at 12:29
  • $\begingroup$ By more information I meant more information about independent variation in things that are naturally correlated. Experimental manipulation is the usual way, but that's probably not possible here. So just interpret the marginal effects. $\endgroup$ – conjugateprior Jul 23 '15 at 12:30
  • $\begingroup$ Thank you for the link and for your answer. Actually I would ask you to have a look to this related question about how reporting the results in the different possible cases of using different models: stats.stackexchange.com/questions/162804/… $\endgroup$ – L_T Jul 24 '15 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.