Differences in nlme output when introducing interactions

Question

I am searching for correlations between a dependent variable and a factor or a combination of factors in a repeated measure design. So I used lme() function in R. However, I am getting very different results depending on whether I add on various factors to the the lme formula as compared to when only one is present. If a factor is found to be significant, shouldn't it remain significant when more factors are introduced in the model?

I give an example of the outputs I get using the two models. In the first model I use one single factor:

library(nlme)
summary(lme(Mode ~ Weight, data = Gravel_ds, random = ~1 | Subject))
Linear mixed-effects model fit by REML
 Data: Gravel_ds 
      AIC      BIC   logLik
  2119.28 2130.154 -1055.64

Random effects:
 Formula: ~1 | Subject
        (Intercept) Residual
StdDev:    1952.495 2496.424

Fixed effects: Mode ~ Weight 
                Value Std.Error DF   t-value p-value
(Intercept) 10308.966 2319.0711 95  4.445299   0.000
Weight        -99.036   32.3094 17 -3.065233   0.007
 Correlation: 
       (Intr)
Weight -0.976

Standardized Within-Group Residuals:
        Min          Q1         Med          Q3         Max 
-1.74326719 -0.41379593 -0.06508451  0.39578734  2.27406649 

Number of Observations: 114
Number of Groups: 19 

As you can see, the p-value for factor Weight is significant. This is the second model, in which I add various factors for searching their correlations:

library(nlme)
summary(lme(Mode ~ Weight*Height*Shoe_Size*BMI, data = Gravel_ds, random = ~1 | Subject))
Linear mixed-effects model fit by REML
 Data: Gravel_ds 
       AIC      BIC    logLik
  1975.165 2021.694 -969.5825

Random effects:
 Formula: ~1 | Subject
        (Intercept) Residual
StdDev:    1.127993 2494.826

Fixed effects: Mode ~ Weight * Height * Shoe_Size * BMI 
                                Value Std.Error DF    t-value p-value
(Intercept)                   5115955  10546313 95  0.4850941  0.6287
Weight                      -13651237   6939242  3 -1.9672518  0.1438
Height                         -18678     53202  3 -0.3510740  0.7487
Shoe_Size                       93427    213737  3  0.4371115  0.6916
BMI                         -13011088   7148969  3 -1.8199949  0.1663
Weight:Height                   28128     14191  3  1.9820883  0.1418
Weight:Shoe_Size               351453    186304  3  1.8864467  0.1557
Height:Shoe_Size                 -783      1073  3 -0.7298797  0.5183
Weight:BMI                      19475     11425  3  1.7045450  0.1868
Height:BMI                     226512    118364  3  1.9136867  0.1516
Shoe_Size:BMI                  329377    190294  3  1.7308827  0.1819
Weight:Height:Shoe_Size          -706       371  3 -1.9014817  0.1534
Weight:Height:BMI                -109        63  3 -1.7258742  0.1828
Weight:Shoe_Size:BMI             -273       201  3 -1.3596421  0.2671
Height:Shoe_Size:BMI            -5858      3200  3 -1.8306771  0.1646
Weight:Height:Shoe_Size:BMI         2         1  3  1.3891782  0.2589
 Correlation: 
                            (Intr) Weight Height Sho_Sz BMI    Wght:H Wg:S_S Hg:S_S Wg:BMI Hg:BMI S_S:BM Wg:H:S_S W:H:BM W:S_S: H:S_S:
Weight                      -0.895                                                                                                    
Height                      -0.996  0.869                                                                                             
Shoe_Size                   -0.930  0.694  0.933                                                                                      
BMI                         -0.911  0.998  0.887  0.720                                                                               
Weight:Height                0.894 -1.000 -0.867 -0.692 -0.997                                                                        
Weight:Shoe_Size             0.898 -0.997 -0.873 -0.700 -0.999  0.995                                                                 
Height:Shoe_Size             0.890 -0.612 -0.904 -0.991 -0.641  0.609  0.619                                                          
Weight:BMI                   0.911 -0.976 -0.887 -0.715 -0.972  0.980  0.965  0.637                                                   
Height:BMI                   0.900 -1.000 -0.875 -0.703 -0.999  0.999  0.999  0.622  0.973                                            
Shoe_Size:BMI                0.912 -0.992 -0.889 -0.726 -0.997  0.988  0.998  0.649  0.958  0.995                                     
Weight:Height:Shoe_Size     -0.901  0.999  0.876  0.704  1.000 -0.997 -1.000 -0.623 -0.971 -1.000 -0.997                              
Weight:Height:BMI           -0.908  0.978  0.886  0.704  0.974 -0.982 -0.968 -0.627 -0.999 -0.975 -0.961  0.973                       
Weight:Shoe_Size:BMI        -0.949  0.941  0.928  0.818  0.940 -0.946 -0.927 -0.751 -0.980 -0.938 -0.924  0.935    0.974              
Height:Shoe_Size:BMI        -0.901  0.995  0.878  0.707  0.998 -0.992 -1.000 -0.627 -0.960 -0.997 -0.999  0.999    0.964  0.923       
Weight:Height:Shoe_Size:BMI  0.952 -0.948 -0.933 -0.812 -0.947  0.953  0.935  0.747  0.985  0.946  0.932 -0.943   -0.980 -0.999 -0.931

Standardized Within-Group Residuals:
        Min          Q1         Med          Q3         Max 
-2.03523736 -0.47889716 -0.02149143  0.41118126  2.20012158 

Number of Observations: 114
Number of Groups: 19 

This time the p-value associated to Weight is not significant anymore. Why? Which analysis should I trust?

In addition, while in the first output the field "value" (which should give me the slope) is -99.036 in the second output it is -13651237. Why are they so different? The one in the first output is the one that seems definitively more reasonable to me.

Answer 1

It's not in general true that "if a factor is found to be significant, [it will] remain significant also when more factors are introduced in the model".

There are quite a few possible reasons for this that are relevant to your problem.

First, it may be helpful to remember the distinction between factors and parameters. For example, Weight is just one factor but is modeled with 1 parameter in model 1 and 8 parameters in model 2. Consequently if you want to see (or test) the effect of Weight in the second model you'll need to explicitly compute the relevant marginal effect, by setting the things it is interacted with to particular values.

In model 1 the relevant marginal effect is given by the parameter value, but that's only because the model is linear and additive. You were expecting that effects would always line up one-to-one with parameters this way, but in models more complicated that your first, they just won't.

Second, your second model has a lot more parameters but just the same amount of data, so it's possible that you don't have enough data to reliably estimate everything that you could estimate if you were to assume a simpler model of the data generating process, such as your first model.

Third, you have variables that are quite tightly linearly related to one another (keyword: collinearity). If you put them all in a model then it may easily be that you can only estimate the effects of combinations of them, e.g. heavy tall people vs light small people. However, the tests that you are looking at the results of are parameter by parameter and do not reflect these combinations. Your significance levels may therefore be all over the place. Those levels are then correct answers to a question you are not actually interested in. In this case you just need more different information to disentangle things.

Finally, if some factors exert their effects through their effects on other factors, then a significant effect of a causally distant effect will be removed when a more causally proximate one is added to the model.

It's worth noting that none of these are a consequence of using mixed effects or nlme. They're just normal issues with regression models.