I am working with the following data:
GROUP (each subject only in one group)
CONDITION (within subject variable, there are 2 conditions (1) baseline - same for all the groups, and (2) experimental - different for each group)
GENDER (of the speaker from the stimuli)
GENDER_S (of the subject)
Item (the audio stimuli)
Response Variable: accent ratings converted to z-scores
Research Question: Will the accent ratings differ between the groups in the experimental condition - and if so, between which? (all the groups listened to the same audio BUT with different visual stimuli)
I have 100 response total per participant = 80 x 100 -> 8000 (40 in one condition and 60 in the other) and model including all these Fixed Effects seems to be the best one based on AIC/BIC (prioritizing BIC in my case), hence in the final model I decided to include both GENDERs.
Since I was first interested only in the group:condition interaction (and which groups differ if any) I used a mixed design ANOVA which showed that it is NOT significant.
I have then moved to LMM, I choose "the best" model which seems to include the genders as well. I tried Anova() on the single model and got SIGNIFICANT group:condition interaction. However, when I did pairwise comparison with glht() and Holm's adjustment there are no significant pairs for group:condition only (averaging on the gender variable). The same happens when I fit a new model with group and condition as the only fixed effect. Also the same outcome even if I use NO adjustment at all.
Any ideas what should one do in a situation like this? Is it OK to ignore ANOVA and use the pairwise comparison since I am interested in which groups differ in the experimental condition (and I need to make sure that they don't in the baseline condition)?