0
$\begingroup$

In my experiment, each participant responded to a set of words. These words could belong to one of four groups (defined by two IVs, each with two levels). There are several different continuous DVs.

When I run simple 2x2 ANOVAs for each DV (leave aside the issue that perhaps I should run a MANOVA first), I find main effects for some DVs, and interactions for others.

However, when I run the analysis using mixed effects regressions (thus keeping the data at the trial level, as opposed to participant means, as with the ANOVA), none of these main effects and/or interactions are significant. I assessed significance using LRT, testing the inclusion of each term into a simpler model. All models included random subject and item intercepts.

Leaving aside issues over the best way to test mixed models, how do I interpret cases where ANOVAs find significant effects but mixed models including random intercepts do not? I assume it has to do with subject/item variance that once contributed to a difference between levels of the independent variables now being modelled as random intercepts?

$\endgroup$

1 Answer 1

2
$\begingroup$

The basic issue is whether or not you are satisfied drawing a conclusion that holds with the specific words in your experiment or whether you want to generalize your results to a larger population of words. If it is the latter, then a significant effect in a model in which "words" is treated as a random effect is required (the mixed model). For some history on this, Google "Language as a fixed effect fallacy."

$\endgroup$
4
  • $\begingroup$ Thanks for the answer! Can you provide a bit of info as to how adding the random effect results in the main effects becoming non-significant? $\endgroup$
    – Dave
    Mar 31, 2017 at 17:06
  • $\begingroup$ To the extent words differ from each other, it is harder to generalize to the population of words. The fixed effect analysis ignores variance among words since it is only concerned with the words in the experiment and not other words. If you didn't want to generalize your results to other subjects then you would ignore variation among subjects. You can find more information here tqmp.org/RegularArticles/vol12-3/p201/p201.pdf $\endgroup$
    – David Lane
    Mar 31, 2017 at 17:50
  • $\begingroup$ Thank you for the paper, it was very informative. Would you be able to explain what's going on "under the hood" in the mixed modal analysis for the Within-Subjects and Between-Stimuli data in Table 3b? There is greater variation among items, and this leads to a non-significant effect when F2 is tested, and when mixed models are used. In the latter example, how exactly does modelling random effects lead to a non-significant effect of the IV? $\endgroup$
    – Dave
    Apr 3, 2017 at 18:26
  • $\begingroup$ When items vary greatly it is hard to generalize to the population of items just as when subjects differ greatly it is hard to generalize to the population of subjects. F2 and mixed models both generalize to the population of items ( mixed models generalize to subjects and items). $\endgroup$
    – David Lane
    Apr 3, 2017 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.