1
$\begingroup$

I am working with the following data:

Fixed effect:

  • GROUP (each subject only in one group)

  • CONDITION (within subject variable, there are 2 conditions (1) baseline - same for all the groups, and (2) experimental - different for each group)

  • GENDER (of the speaker from the stimuli)

  • GENDER_S (of the subject)

Random Effects:

  • Subject

  • Item (the audio stimuli)

Response Variable: accent ratings converted to z-scores

Research Question: Will the accent ratings differ between the groups in the experimental condition - and if so, between which? (all the groups listened to the same audio BUT with different visual stimuli)

I have 100 response total per participant = 80 x 100 -> 8000 (40 in one condition and 60 in the other) and model including all these Fixed Effects seems to be the best one based on AIC/BIC (prioritizing BIC in my case), hence in the final model I decided to include both GENDERs.

Since I was first interested only in the group:condition interaction (and which groups differ if any) I used a mixed design ANOVA which showed that it is NOT significant.

I have then moved to LMM, I choose "the best" model which seems to include the genders as well. I tried Anova() on the single model and got SIGNIFICANT group:condition interaction. However, when I did pairwise comparison with glht() and Holm's adjustment there are no significant pairs for group:condition only (averaging on the gender variable). The same happens when I fit a new model with group and condition as the only fixed effect. Also the same outcome even if I use NO adjustment at all.

Any ideas what should one do in a situation like this? Is it OK to ignore ANOVA and use the pairwise comparison since I am interested in which groups differ in the experimental condition (and I need to make sure that they don't in the baseline condition)?

Improve this question
$\endgroup$
12
  • $\begingroup$ Is this contrast between significant anova and non-significant post-hoc tests the problem? "I tried Anova() on the single model and got SIGNIFICANT group:condition interaction. However, when I did pairwise comparison with glht() and Holm's adjustment there are no significant pairs for group:condition only (averaging on the gender variable)." For this see stats.stackexchange.com/a/352612/164061 and the links. $\endgroup$
    – Sextus Empiricus
    Commented Oct 27, 2018 at 7:21
  • $\begingroup$ Possible duplicate of R Tukey HSD Anova: Anova significant, Tukey not? and How can I get a significant overall ANOVA but no significant pairwise differences with Tukey's procedure? $\endgroup$
    – Sextus Empiricus
    Commented Oct 27, 2018 at 7:42
  • $\begingroup$ Thank you! I am just wandering, this seems to be more related to Tukey than Holm (which I used) or am I looking at it the wrong way. $\endgroup$
    – Karvin
    Commented Oct 27, 2018 at 7:46
  • $\begingroup$ It is just a different adjustment, but the principle remains the same. For the same anova result you can have different distributions of the pairwise differences which may or may not turn out to be significant in pairwise comparisons. $\endgroup$
    – Sextus Empiricus
    Commented Oct 27, 2018 at 7:48
  • $\begingroup$ Thank you Martijn. Any idea what is the common practice in a situation like this? Ignore ANOVA? $\endgroup$
    – Karvin
    Commented Oct 27, 2018 at 7:49

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.