0
$\begingroup$

I know that the question has been submitted several times but given my level of knowledge, I'm afraid I haven't been able to find a satisfactory answer. I have two fixed effects : frequency (high vs low) and predictability (high vs low). my model is as follows :

m1=lmer(log(FF) ~ frequence * Predic + (1 | Sujet) + (1 | item), data= FreqPredicExpert72)

The anova function returns : 

Type III Analysis of Variance Table with Satterthwaite's method
                  Sum Sq Mean Sq NumDF  DenDF F value    Pr(>F)    
Predic           1.78800 1.78800     1 68.035 23.1208 8.774e-06 ***
frequence        0.78005 0.78005     1 68.155 10.0869  0.002243 ** 
Predic:frequence 0.16368 0.16368     1 67.894  2.1166  0.150321   

The summary function returns : 


Linear mixed model fit by REML. t-tests use Satterthwaite's method ['lmerModLmerTest']
Formula: log(FF) ~ Predic * frequence + (1 | Sujet) + (1 | item)
   Data: FreqPredicExpert72
REML criterion at convergence: 558.9
Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.6928 -0.5517 -0.0021  0.5770  4.3347 
Random effects:
 Groups   Name        Variance Std.Dev.
 item     (Intercept) 0.002058 0.04536 
 Sujet    (Intercept) 0.014550 0.12062 
 Residual             0.077333 0.27809 
Number of obs: 1580, groups:  item, 72; Sujet, 28
Fixed effects:
                     Estimate Std. Error       df t value Pr(>|t|)    
(Intercept)           5.33733    0.02883 49.82410 185.118   <2e-16 ***
PredicHP             -0.05921    0.02490 67.27315  -2.378   0.0202 *  
frequenceHF          -0.03042    0.02489 67.28622  -1.222   0.2260    
PredicHP:frequenceHF -0.05133    0.03528 67.89410  -1.455   0.1503    

Correlation of Fixed Effects:
            (Intr) PrdcHP frqnHF
PredicHP    -0.428              
frequenceHF -0.429  0.497       
PrdcHP:frHF  0.302 -0.705 -0.705

There is no difference concerning p value fonte interaction, but as on can see, anova returns a significant effect of frequency (p = 0.002243 ** ) while this effect is not significant in results with summary (p = 0.2260). For me it's a big problem.

My question is threefold

1) Why both p value of the interaction and p value of the Predic effect are identical in summary() and in anova() function? And why it is not the case for the frequency effect?

2) In what outcome can we trust? (Anova or summary?). Pratically, effect of frequency amounted to 6 ms in the group of 21 adults. It seems unlikely that this effect is significant. So I will tend to rely on results from the summary() rather than anova()

3) What I have just described is part of my data. IN fact, I have three groups of participants (grade 3, grade 5 and Adults).

To analyse my data, I conducted a 3 way-anova : Group x frequency x Predic The model used is as follows :

m1=lmer(log(FF) ~ Group * frequence * Predic + (1 | Sujet) + (1 | item), data= FreqPredicExpert72)

For this model, anova() and summary() returns very different results. If I understand correctly, it’s because summary() function tests contrasts? Is it possible to specify something so that summary function gives global effect for the Group x frequency x Predic interaction as in anova() function. Does this have anything to do with contrast specification?

$\endgroup$
5
  • $\begingroup$ i apologize for the the table which are not formatted correctly $\endgroup$
    – rach
    Jun 4, 2020 at 21:43
  • $\begingroup$ Is there some reason you are using Type III Anova? And just how was Type III defined in this implementation? See this question about these difficulties. Also, the order of entry of predictors into the model can matter in some Types of anova(). In the first model with "Type II!" Anova you entered frequence * Predic but in the model reported by Summary it was in the order Predic * frequency. Check if that makes a difference for the anova() result. (Order shouldn't affect the Summary result.) $\endgroup$
    – EdM
    Jun 5, 2020 at 14:39
  • $\begingroup$ Thank you for the answer. I have no reason to use Type III Anova. Changing the order ( frequency * Predict or Predic* frequency) have no effect on p value in anova or in summary function. Using car::Anova(time.lm, type = 3) gives the Sams p values as the summary() function. Do you think that the use of summary is better than the use of anova given that car::Anova(time.lm, type = 3) and the Anova(m1, type=c("III")) give same results as summary() function? Thank you. $\endgroup$
    – rach
    Jun 5, 2020 at 21:26
  • $\begingroup$ Just to be clear: the first anova() function presented in your question was from the lmerTest package. Working on a single object produced by lmer(), does that function (with its default type="III" setting) give different results from Anova() (with a type=3 setting) in the car package? Also, are your objects time.lm and m1 the same? It would be best to clarify those issues by editing the question itself, to make it easier for others to find the information. Also, comments can sometimes be lost. $\endgroup$
    – EdM
    Jun 5, 2020 at 22:07
  • $\begingroup$ I'll try to be clear, the problem concerns the p value for the factor "frequence" : 1) P value is significant (p<.001)when i use anova(m1), anova(m1, type = 2) or car::Anova(m1) 2) P value becomes NOT significant ( p = 0.22) when i use Anova(m1, type=c("III")) or summary() function So, what is the most likely outcome, given that inspection of the data shows a very limited effect of frequence. $\endgroup$
    – rach
    Jun 6, 2020 at 11:49

1 Answer 1

0
$\begingroup$

This apparent discrepancy comes from the difficulty in interpreting "main effects" (e.g., the coefficient for frequence reported by summary(), or a Type III Anova test) when there is an interaction.

With the standard treatment coding default in R, the significance test for frequence reported by summary() in your interaction model will examine whether the coefficient is different from 0 when predictability is at its reference level. It looks like the reference levels for frequence and predictability are both set at "low" in your model. As a concrete example say that there is a significant interaction between the two predictors, with 0 influence of frequence on outcome when predictability is "low" but a large influence when predictability is "high".

Then the coefficient reported by summary() for frequence should be insignificant, with a coefficient of 0! That's because at the reference level of predictability there is no influence of frequence on outcome. That doesn't mean that frequence is unimportant. You would still have a large interaction coefficient involving it. And if you simply set the reference level for predictability to "high" then the coefficient reported by summary() for frequence would now be significantly different from 0.

So you have to be very circumspect when interpreting apparent "main effects" when predictors are included in interactions. As another example, just centering a continuous predictor can change the apparent "significance" of individual coefficients for all predictors interacting with it.

In your case there is an interaction coefficient (-0.05133) of magnitude similar to that of the "main effect" of predictability (-0.05921, for the case of "low" frequence), even though it didn't reach the magical threshold of 5% significance. The combination of the interaction term and the "main effect" of frequence (-0.03042) would estimate an effect of frequence at "high" predictability of -0.08175, which might well pass that threshold. Try switching the reference level of predictability and see for yourself.

This difficulty with interpreting the results of summary() will only be compounded in your final 3-way interaction model. The "main effects" for each predictor would be when both of the others are at their reference levels, however those were chosen.

Type III Anova can lead to similar difficulties in interpretation, as the significance of frequence will be assessed after the contributions of both predictability and the interaction between frequence and predictability are evaluated. So an "insignificant" effect of frequence doesn't mean it isn't important, just that its "main effect" is dwarfed in part by its interaction with predictability. As @gung puts it in this answer, there is an inherent problem with Type III Anova that can lead to false negative results:

If you sum the type III [sum of squares] SS in an ANOVA table, you will notice that they do not equal the total SS. In other words, this analysis must be wrong, but errs in a kind of epistemically conservative way.

Or, as the Warning in the manual page for the car Anova() function puts it:

Be careful of type-III tests.

In your particular case, one might argue that the lack of a statistically significant interaction term (at least in the 2-predictor model) could allow you to revert to a simple additive model. Then this problem would go away, although you should report that you first tested the interaction model.

What you presumably care most about is whether frequence is associated with outcome whether or not that association is primarily due to an interaction with predictability. With fixed-effect models the rms package in R handles this nicely by doing a Wald-type test of whether all of the coefficients (up to 2-way interactions) involving a predictor are 0, taking into account the covariances among the coefficient estimates. I don't work much with mixed models but there should be ways to do the same analysis for those. That might be how Anova() in car performs its Wald tests (as opposed to its default likelihood-ratio tests), but I'm not sure. You might also consider the emmeans package as a tool for evaluating effects of interest in mixed models with interaction terms.

What I'm still confused about is why the "Type III" test reported by anova() from the lmerTest package gave a significant coefficient for frequence while the other Type III tests didn't. But that's seems to be more a question of software implementation than a statistical issue.

$\endgroup$
9
  • $\begingroup$ Thank you very much for this detailed explanation. I think that I better understand my problem. As proposed, I will try to use emmeans package to evaluate in particular effect of frequency. Many thanks again for your response. $\endgroup$
    – rach
    Jun 6, 2020 at 14:30
  • $\begingroup$ @rach just remember that with interaction terms there is no single "effect of frequency." Its "effect" depends on the levels of all the other predictors interacting with it. $\endgroup$
    – EdM
    Jun 6, 2020 at 15:47
  • $\begingroup$ So as I understand it, if I'm interesting in the single "effect of frequency", I should test it in a model as m1=lmer(log(FF) ~ frequence + Predic + (1 | Sujet) + (1 | item), data= FreqPredicExpert72), where the interaction terms does not appears? $\endgroup$
    – rach
    Jun 6, 2020 at 16:21
  • $\begingroup$ @rach yes, but if there is truly an interaction there is no "single effect of frequency." For example, in an observational study with a true interaction, the apparent "single effect of frequency" would depend on the relative number of cases having Predic="low" and Predic="high". Those are the types of problems that emmeans can help with. In your particular case with an insignificant interaction term you could justify proceeding that way, but you might miss finding something important that further study could elucidate. $\endgroup$
    – EdM
    Jun 6, 2020 at 16:27
  • $\begingroup$ Ok @EDM, All this information is very formative and useful. In fact, I have several groups, some in which frequency x Predic interaction is significant, other not (I do this because I have initially a significant interaction between Groups, frequency and Predic). So, according to the p value of the interaction, I can or not focus on single/main effects. I will try to use emmeans function and post the outcome. Very pleased with your advice! $\endgroup$
    – rach
    Jun 6, 2020 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.