I am conducting a mixed design ANOVA test using ezANOVA. I ran two different tests as follows:

Test1: within subject variable w, between subject variables age and x1.

Test2: within subject variable w, between subject variables age and x2.

Both tests are on the same data and same dependent variable (but I built two separate models), all independent variables are categorical.

But the output shows that, in Test1, the main effect of age is not significant, but in Test2 it is significant. My question is if this can happen or I am doing something wrong? In both tests, interaction between age and x1/x2 are not significant.

  • 1
    $\begingroup$ Maybe need to write down the model mathematically. Did you fit two models and perform one test on each of them, or fit one model and test two hypotheses based on the same model? $\endgroup$ – user158565 Oct 10 '18 at 1:16
  • $\begingroup$ I had two separate models $\endgroup$ – Rakib Oct 10 '18 at 4:08
  • 1
    $\begingroup$ Could you write down you two models mathematically? $\endgroup$ – user158565 Oct 10 '18 at 21:46

This is a common occurrence, and I'm sure there has to be a good explanation on Cross Validated somewhere. But I'll mention a couple of things.

  • Adding terms uses up denominator degrees of freedom in the F test for the other terms. In your case, if x1 is a factor variable with more levels than x2, the sums of squares for Age could be the same for both models, but the F-test could be different based on the denominator degrees of freedom.
  • Correlated independent variables. When independent variables are correlated, the shared sum of squares may be assigned to only one term, or only the unique sum of squares for a term may be counted. I think it's always a good idea to check the correlations among all independent variables, and between the dependent variable and each independent variable, as a preliminary step.
  • Types of sums of squares can matter. You probably know that stats::anova uses type I sum of squares. car::Anova uses type II by default. I don't know what type ezANOVA uses by default. In any case, using these different types affects how the sums of squares are apportioned to terms, and so affects the F-tests.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.