0
$\begingroup$

I am struggling to interpret my output in R. It does not make sense to me.

I first regressed participants' ratings (= value) on manipulations (= variable: H high, AI high, H low, AI low, and control). I then checked whether this effect is moderated by the participants' experience (= Frequency :7 groups, the higher, the less experience). Frequency was added as an interaction factor in my model:

H4_mod <- lmer(value ~ variable * Frequency + (1|subject), data = 
  ANALYSIS_long, REML = FALSE)

Then, in order to find out whether the interaction effect is statistically significant, I compared the H4_mod with a reduced model:

red_H4_mod <- lmer(value ~ variable + Frequency + (1|subject), data = 
  ANALYSIS_long, REML = FALSE)

An ANOVA and a look at cAICs revealed that the interaction is statistically significant. So far, so good.

Now, I want to find out what the effect looks like. I predicted that the higher the experience, the lower the effect of the prediction (value ~ variable).

I get confused with the output I get from

summary(H4_mod)

Fixed effects:
                               Estimate Std. Error         df t value Pr(>|t|)    
    (Intercept)               6.846e+00  2.872e-01  8.334e+02  23.839  < 2e-16 ***
    variable[T.2]             9.542e-01  3.223e-01  1.032e+03   2.960 0.003145 ** 
    variable[T.3]            -8.000e-01  3.223e-01  1.032e+03  -2.482 0.013226 *  
    variable[T.4]             1.104e+00  3.223e-01  1.032e+03   3.426 0.000638 ***
    variable[T.5]            -1.154e+00  3.223e-01  1.032e+03  -3.581 0.000359 ***
    Frequency2                3.310e-01  3.739e-01  8.334e+02   0.885 0.376362    
    Frequency3                5.490e-01  3.650e-01  8.334e+02   1.504 0.132893    
    Frequency4               -5.164e-01  5.615e-01  8.334e+02  -0.920 0.358013    
    Frequency5                2.875e-01  6.421e-01  8.334e+02   0.448 0.654468    
    Frequency6                5.417e-02  6.043e-01  8.334e+02   0.090 0.928602    
    Frequency7               -1.156e+00  5.295e-01  8.334e+02  -2.183 0.029329 *  
    variable[T.2]:Frequency2 -3.397e-01  4.197e-01  1.032e+03  -0.809 0.418551    
    variable[T.3]:Frequency2 -4.087e-01  4.197e-01  1.032e+03  -0.974 0.330432    
    variable[T.4]:Frequency2 -5.592e-01  4.197e-01  1.032e+03  -1.332 0.183033    
    variable[T.5]:Frequency2 -2.545e-01  4.197e-01  1.032e+03  -0.606 0.544378    
    variable[T.2]:Frequency3 -5.901e-01  4.097e-01  1.032e+03  -1.440 0.150082    
    variable[T.3]:Frequency3  5.128e-03  4.097e-01  1.032e+03   0.013 0.990015    
    variable[T.4]:Frequency3 -4.837e-01  4.097e-01  1.032e+03  -1.181 0.238045    
    variable[T.5]:Frequency3  2.875e-01  4.097e-01  1.032e+03   0.702 0.482980    
    variable[T.2]:Frequency4  3.407e-02  6.303e-01  1.032e+03   0.054 0.956904    
    variable[T.3]:Frequency4  2.471e-01  6.303e-01  1.032e+03   0.392 0.695154    
    variable[T.4]:Frequency4  2.841e-01  6.303e-01  1.032e+03   0.451 0.652301    
    variable[T.5]:Frequency4  1.025e+00  6.303e-01  1.032e+03   1.626 0.104285    
    variable[T.2]:Frequency5 -5.708e-01  7.208e-01  1.032e+03  -0.792 0.428548    
    variable[T.3]:Frequency5 -2.250e+00  7.208e-01  1.032e+03  -3.122 0.001848 ** 
    variable[T.4]:Frequency5 -1.288e+00  7.208e-01  1.032e+03  -1.786 0.074342 .  
    variable[T.5]:Frequency5 -1.413e+00  7.208e-01  1.032e+03  -1.960 0.050295 .  
    variable[T.2]:Frequency6 -1.970e-01  6.783e-01  1.032e+03  -0.290 0.771525    
    variable[T.3]:Frequency6 -9.714e-01  6.783e-01  1.032e+03  -1.432 0.152417    
    variable[T.4]:Frequency6 -3.327e-01  6.783e-01  1.032e+03  -0.491 0.623863    
    variable[T.5]:Frequency6 -1.203e+00  6.783e-01  1.032e+03  -1.773 0.076448 .  
    variable[T.2]:Frequency7  3.658e-01  5.944e-01  1.032e+03   0.616 0.538350    
    variable[T.3]:Frequency7 -6.000e-02  5.944e-01  1.032e+03  -0.101 0.919610    
    variable[T.4]:Frequency7  2.583e-02  5.944e-01  1.032e+03   0.043 0.965340    
    variable[T.5]:Frequency7  4.742e-01  5.944e-01  1.032e+03   0.798 0.425178    
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Can someone explain it to me? The intercept is my control manipulation ("variable"), but what happened to Frequency 1? I read that only the Frequency 7 group has a significant impact on the relationship (value ~ variable). When I look further down at the (what I think) actual interaction effects, these are only significant for Frequency 5.

Furthermore, I tried to figure out the relationships by looking at an effects plot

e <- allEffects(H4_mod)
plot(e, multiline = TRUE, confint = TRUE, ci.style = "bars")

Plot for the effects retrieved from the effects package

Unfortunately, this confuses me even more... Can someone help me interpreting the results? So far, I can only say that Frequency significantly interacts with the value ~ variable relationship. But I would love (and have) to make a statement about the direction.

THANK YOU!

$\endgroup$
  • $\begingroup$ You have used the effects tag which refers to a specific R package (not to effects in general). If the tag does not belong here, I suggest to remove it. $\endgroup$ – Richard Hardy May 14 at 10:04
  • $\begingroup$ Hej, thank you! For the effects plot, I used the effects package :) $\endgroup$ – Theresa May 14 at 10:14
  • $\begingroup$ OK, then my comment is irrelevant. $\endgroup$ – Richard Hardy May 14 at 10:30
0
$\begingroup$

This is really quite simple, the only issue is that you've fallen into the trap of categorizing ordinal values simply because they are ordinal. It's not surprising that the the global test of (5-1) * (7-1) = 24 fixed effects representing the high dimensional interaction is statistically significant, the chance for overfitting and small sample bias is high. Even the plot, which in most cases would be an intelligent summary of such high dimensional output (because it shows Frequency continuously), shows massively overlapping CI bounds.

If you want a simple and easy to interpret output, consider fitting the Frequency and Variable measures as numeric instead. Be sure that the numeric value actually corresponds to the observed level. This gives you a more powerful and more interpretable test.

$\endgroup$
  • $\begingroup$ Hi AdamO, thank you! I will try that! $\endgroup$ – Theresa May 15 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.