2
$\begingroup$

I am new to this. My study has three conditions (between subjects - low coordination, high coordination, high coordination with ostensive cues) and three repetitions of a game (within subjects - Game 1, Game 2, Game 3). The outcome variable is binary (i.e., participant's left the game, they did not leave the game).

I've carried out 3 GLMERs using the following code:

S1_glmer <- glmer(left_game ~ condition + game + condition:game + (1 | participant),
                  data = S1_data,
                  family = binomial(link="logit"),control = glmerControl(optimizer = "bobyqa"),
                  nAGQ = 1)

See all three outputs below

The condition estimates on the bottom two glmers add up, i.e, glmer 2, intercept for high coordination with ostensive cues is 9.1848 - minus -0.9203 for high coordination = 8.2645 which is the intercept for high coordination in glmer 3. But the condition estimates for glmer 1 do not add up to those for glmer 2 or glmer 3. According to the p-values in glmer 1, there is no significant condition difference between low coordination and the other conditions. But in glmer 2 and 3, there are significant condition differences. Can anyone explain what's happening here?

I changed the reference level to find out whether there was a significant difference between the 2nd and 3rd conditions: high coordination and high coordination with ostensive cues.

Now I feel like I can't trust the original results.

Glmer 1) Output when reference level is low coordination, Game 1:

Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) [glmerMod]
Family: binomial  ( logit )
Formula: left ~ cond + game + cond:game + (1 | participant)
Data: S1_data
Control: glmerControl(optimizer = "bobyqa")

AIC      BIC   logLik deviance df.resid 
176.7    209.8    -78.3    156.7      192 

Scaled residuals: 
Min      1Q  Median      3Q     Max 
-1.7321 -0.0004  0.0001  0.0101  4.9872 

Random effects:
Groups      Name        Variance Std.Dev.
participant (Intercept) 723.7    26.9    
Number of obs: 202, groups:  participant, 72

Fixed effects:
Estimate     Std. Error   z value   Pr(>|z|)    
(Intercept) 8.2474        1.9877     4.149  3.34e-05 ***
conditionHigh coordination 0.8632        2.2934     0.376   0.70664 
conditionHigh coordination with ostensive cues 1.8618        2.7681 0.673   0.50121 
gameGame 2                                     10.6282       2.3868     4.453   8.47e-06 ***
gameGame 3                                     14.3519       5.8753 2.443   0.01458 *
conditionHigh coordination:gameGame2           1.0957        3.4069 0.322   0.74774 
conditionHigh coordination with ostensive cues:gameGame 2  -8.5113 2.8270 -3.011  0.00261 **
conditionHigh coordination:gameGame 3         -2.8912       5.9182  -0.489    0.62518   
conditionHigh coordination with ostensive cues:gameGame 3 -10.1465 6.0885 -1.666  0.09561 .

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
            (Intr) cndHgc cHcwoc gamGm2 gamGm3 cHc:G2 ccwoc2 cHc:G3
cndHghcrdnt -0.621                                                 
cndHghcrwoc -0.494  0.421                                          
gameGame 2   0.252  0.035  0.050                                   
gameGame 3   0.304  0.002  0.024  0.583                            
cndHcrdn:G2  0.070 -0.100 -0.043 -0.446 -0.142                     
cndHcwoc:G2 -0.188 -0.032 -0.086 -0.819 -0.466  0.377              
cndHcrdn:G3 -0.159 -0.046 -0.028 -0.431 -0.837  0.280  0.348       
cndHcwoc:G3 -0.276 -0.004 -0.040 -0.545 -0.946  0.137  0.533  0.796

Glmer 2) Output when reference level is high coordination with ostensive cues, Game 1:

Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) [glmerMod]
Family: binomial  ( logit )
Formula: left ~ cond + game + cond:game + (1 | participant)
Data: S1_data
Control: glmerControl(optimizer = "bobyqa")

AIC      BIC   logLik deviance df.resid 
180.1    213.1    -80.0    160.1      192 

Scaled residuals: 
Min      1Q  Median      3Q     Max 
-1.6992 -0.0009  0.0001  0.0153  4.6034 

Random effects:
Groups      Name        Variance Std.Dev.
participant (Intercept) 425.6    20.63   
Number of obs: 202, groups:  participant, 72

Fixed effects:
                               Estimate     Std. Error  z value  Pr(>|z|)       
(Intercept)                      9.1848         2.1924      4.189    2.80e-05   ***
condHigh coordination           -0.9203         2.3928      -0.385   0.700510
condLow coordination            -16.0097        3.6128          -4.431   9.36e-06   ***
gameGame 2                        1.9997        1.5684      1.275    0.202292       
gameGame 3                         4.0240       1.9118      2.105    0.035304   *
condHigh coordination:gameGame 2   8.4752       3.1819      2.664    0.007732   **
condLow coordination:gameGame 2   12.9762       3.5373      3.668    0.000244   ***
condHigh coordination:gameGame 3  6.1720        3.4743      1.776    0.075656   .
condLow coordination:gameGame 3   13.8269       5.5511      2.491    0.012745   *

Correlation of Fixed Effects:
            (Intr) cHcwoc cndLwc gamGm2 gamGm3 ccwoc2 cLc:G2 ccwoc3
cndHghcrwoc -0.483                                                 
cndLwcrdntn -0.738  0.251                                          
gameGame 2   0.231  0.060 -0.301                                   
gameGame 3   0.221  0.057 -0.287  0.498                            
cndHcwoc:G2 -0.178 -0.100  0.237 -0.870 -0.422                     
cndLcrdn:G2  0.158 -0.037 -0.436 -0.499 -0.113  0.456              
cndHcwoc:G3 -0.151 -0.084  0.203 -0.397 -0.835  0.475  0.111       
cndLcrdn:G3  0.249 -0.020 -0.457  0.042 -0.260 -0.020  0.485  0.252

Glmer 3) Output when reference level is high coordination, Game 1:

Generalized linear mixed model fit by maximum likelihood (Laplace
Approximation) [glmerMod]
Family: binomial  ( logit )
Formula: left ~ cond + game + cond:game + (1 | participant)
Data: S1_data
Control: glmerControl(optimizer = "bobyqa")

AIC      BIC   logLik deviance df.resid 
180.1    213.1    -80.0    160.1      192 

Scaled residuals: 
Min      1Q  Median      3Q     Max 
-1.6992 -0.0009  0.0001  0.0153  4.6034 

Random effects:
Groups      Name        Variance Std.Dev.
participant (Intercept) 425.6    20.63   
Number of obs: 202, groups:  participant, 72

Fixed effects:
                                            Estimate    Std. Error  z value Pr(>|z|)        
(Intercept)                                     8.2645          1.7992           4.594  4.36e-06    ***                                                                                            
condHigh coordination with ostensive cues       0.9203          2.3926           0.385  0.700490
condLow coordination                           -15.0893         3.3701           -4.477 7.55e-06    ***
gameGame 2                                      10.1959         2.8428            3.685 0.000229    ***
gameGame 3                                      10.4749         3.0173            3.379 0.000727    ***
condHigh coordination with ostensive cues:gameGame 2  4.5010    3.1753          -2.669  0.007605    **
condLow coordination:gameGame 2                 -8.4752         3.5680           1.262  0.207126        
condHigh coordination with ostensive cues:gameGame 3  -6.1720   3.4707         -1.778  0.075355 .
condLow coordination:gameGame 3                  7.6549         5.2865           1.448  0.147617        

Correlation of Fixed Effects:
            (Intr) cHcwoc cndLwc gamGm2 gamGm3 ccwoc2 cLc:G2 ccwoc3
cndHghcrwoc -0.483                                                 
cndLwcrdntn -0.738  0.251                                          
gameGame 2   0.231  0.060 -0.301                                   
gameGame 3   0.221  0.057 -0.287  0.498                            
cndHcwoc:G2 -0.178 -0.100  0.237 -0.870 -0.422                     
cndLcrdn:G2  0.158 -0.037 -0.436 -0.499 -0.113  0.456              
cndHcwoc:G3 -0.151 -0.084  0.203 -0.397 -0.835  0.475  0.111       
cndLcrdn:G3  0.249 -0.020 -0.457  0.042 -0.260 -0.020  0.485  0.252
```
Improve this question
$\endgroup$
9
  • 2
    $\begingroup$ tl;dr whenever you have an interaction in your model, changing the zero point/reference level for some predictors will change the meaning, and hence the p-value, of the main effects of other predictors that are involved in the interaction. en.wikipedia.org/wiki/Principle_of_marginality $\endgroup$
    – Ben Bolker
    Commented Oct 10, 2023 at 1:15
  • $\begingroup$ If a predictor X participates in interactions, you can't interpret the "main effect" of X meaningfully on its own. The reason is that we can't apply the recipe: "the interpretation of $\beta$ is the change in the outcome Y when we change X while keeping the other predictors fixed". We can't vary the main effect while keeping the interaction fixed. What comparisons do you want to make? It's worth learning how to use the emmeans package. $\endgroup$
    – dipetkov
    Commented Oct 10, 2023 at 9:10
  • $\begingroup$ Thanks, to editors, migrators and responders. @dipetkov thanks to you and Ben, I understand a bit more about the impact of the interactions now. Game is a bit of a nuisance variable, but I had to include it because of task fatigue over time. I was interested in the impact of the conditions on whether participants left the game. There were no sig diffs between the low coordination condition and the other conds for this outcome measure. However, for completeness, I was hoping to test whether there was a sig diff between the other two conds. How should I do this (instead of what I've done here)? $\endgroup$
    – Melissa D. Perring
    Commented Oct 11, 2023 at 21:42
  • $\begingroup$ @BenBolker, thanks for your response. $\endgroup$
    – Melissa D. Perring
    Commented Oct 11, 2023 at 21:43
  • $\begingroup$ You can get the three odds ratios with pairs(emmeans(fit3, ~ condition | game, type = "response")). This calculates the odds ratios at each level of game. From the phrase "task fatigue over time" I suspect that the third game is the last game played. So it may be most relevant to focus on the final set of odds ratios. Then you'd use pairs(emmeans(fit3, ~ condition | game, at = list(game = "3"), type = "response")). $\endgroup$
    – dipetkov
    Commented Oct 11, 2023 at 22:09

1 Answer 1

Reset to default
1
$\begingroup$

These comments address why the p-values change and the estimates don't add up:

  1. whenever you have an interaction in your model, changing the zero point/reference level for some predictors will change the meaning, and hence the p-value, of the main effects of other predictors that are involved in the interaction. en.wikipedia.org/wiki/Principle_of_marginality – Ben Bolker 2 days ago

  2. If a predictor X participates in interactions, you can't interpret the "main effect" of X meaningfully on its own. The reason is that we can't apply the recipe: "the interpretation of β is the change in the outcome Y when we change X while keeping the other predictors fixed". We can't vary the main effect while keeping the interaction fixed. – dipetkov 2 days ago

These comments address how to compare all three conditions to each other (NB. within each condition, there are 3 games in chronological order - there may be task fatigue over time):

It's worth learning how to use the emmeans package: emmeans interactions

You can get the three odds ratios with pairs(emmeans(fit3, ~ condition | game, type = "response")). This calculates the odds ratios at each level of game.

From the phrase "task fatigue over time" I suspect that the third game is the last game played. So it may be most relevant to focus on the final set of odds ratios. Then you'd use pairs(emmeans(fit3, ~ condition | game, at = list(game = "3"), type = "response")).

The odds ratios estimates are parametrization independent: you'll get the same answer with any of the three fitted model which are actually the same model, with parameters coded differently but equivalently. – dipetkov 21 hours ago

These videos helped me understand emmeans: intro interactions

This blog was also helpful: getting started with emmeans

Improve this answer
$\endgroup$
10
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Oct 12, 2023 at 20:07
  • $\begingroup$ Thanks for posting; to be more specific, my idea was that rather than cutting and pasting comments, you would compose a complete answer based on the comments ... $\endgroup$
    – Ben Bolker
    Commented Oct 13, 2023 at 19:45
  • 1
    $\begingroup$ @BenBolker, apologies, I'm new here, I'll write something more coherent at the weekend, once I've sorted the gmmeans. $\endgroup$
    – Melissa D. Perring
    Commented Oct 13, 2023 at 21:36
  • $\begingroup$ Good start at writing an answer. Here is also the Interaction vignette in the emmeans documentation. (And I apologize for writing comments rather than an answer; that turned out to not be very helpful.) $\endgroup$
    – dipetkov
    Commented Oct 14, 2023 at 13:25
  • $\begingroup$ @dipetkov, thank you, all your help is much appreciated! $\endgroup$
    – Melissa D. Perring
    Commented Oct 14, 2023 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.