I have two groups of 10 participants who were assessed three times during an experiment. To test for differences between groups and across the three assessments, I ran a 2x3 mixed design ANOVA with group (control, experimental), time (first, second, three), and group x time. Both time and group resulted significant, besides there was a significant interaction group x time.

I don't know very well how to proceed to further check for the differences between the three times of assessments, also respect to group membership. In fact, at the beginning I only specified in the options of the ANOVA to compare all the main effects, using the Bonferroni's correction. However, then I realized that this way they were compared the differences in time of the total sample, without group distinction, am I right?

Therefore, I searched a lot on the internet to find a possible solution, but with scarce results. I only found 2 cases similar to mine, but their solutions are opposite!

  1. In an article, after the mixed design, the authors ran 2 repeated measures ANOVA as a post-hoc, one for each group of subjects. This way, the two groups are analysed separately without any correction, am I right?
  2. In a guide on the internet, they say to add manually in the SPSS syntax COMPARE(time) ADJ(BONFERRONI), just after /EMMEANS=TABLES(newgroup*time), while running the mixed ANOVA. This way, the three times are compared separately for each group, with Bonferroni correction, am I right?

What do you think? Which would be the correct way to proceed?

  • 1
    $\begingroup$ stats.stackexchange.com/questions/575/… $\endgroup$ Mar 6, 2013 at 5:17
  • $\begingroup$ Winer (1962)'s master text on stats. provides formulae for the errors terms to be used in post hoc comparisons following many kinds of ANOVA, including this one. $\endgroup$
    – user41999
    Mar 16, 2014 at 0:36
  • $\begingroup$ Hi @StuartMcKelvie, could you give more details? As it stands, your answer is hardly usable by the OP or future visitors. (Plus, you don't provide a reference for Winer [1962], and since it's so old, it might not be easy to find.) $\endgroup$ Mar 16, 2014 at 1:03
  • $\begingroup$ I found this free chapter from IBM SPSS Statistics (18&19): Workbook psychtestingonline.com/PDFDownloader.aspx?pdf=3 It's exactly about your case $\endgroup$
    – sviter
    Aug 4, 2014 at 14:06
  • $\begingroup$ I'm having exactly the same issue. Did you decide on a particular method in the end? Thank you. $\endgroup$ Sep 26, 2016 at 19:50

3 Answers 3


Answer edited to implement encouraging and constructive comment by @Ferdi

I would like to:

  1. provide an answer with a full contained script
  2. mention one can also test more general custom contrasts using the /TEST command
  3. argue this is necessary in some cases (ie the EMMEANS COMPARE combination is not enough)

I assume to have a database with columns: depV, Group, F1, F2. I implement a 2x2x2 mixed design ANOVA where depV is the dependent variable, F1 and F2 are within subject factors and Group is a between subject factor. I further assume the F test has revealed that the interaction Group*F2 is significant. I therefore need to use post hoc t-tests to understand what drives the interaction.

MIXED depV BY Group F1 F2 
  /FIXED=Group F1 F2 Group*F1 Group*F2 F1*F2 Group*F1*F2 |  SSTYPE(3) 
  /EMMEANS=TABLES(Group*F2) COMPARE(Group) ADJ(Bonferroni)
  /TEST(0) = 'depV(F2=1)-depV(F2=0) differs between groups' 
    Group*F2 1/4 -1/4 -1/4 1/4 
    Group*F1*F2 1/8 -1/8 1/8 -1/8 -1/8 1/8 -1/8 1/8 
  /TEST(0) = 'depV(Group1, F2=1)-depV(Group2, F2=1)' Group 1 -1
    Group*F1 1/2 1/2 -1/2 -1/2 
    Group*F2 1 0 -1 0  
    Group*F1*F2 1/2 0 1/2 0 -1/2 0 -1/2 0 .

In particular the second t-test corresponds to the one performed by the EMMEANS command. The EMMEANS comparison could reveal for example that depV was bigger in Group 1 on the condition F2=1.

However the interaction could also be driven by something else, which is verified by the first test: the difference depV(F2=1)-depV(F2=0) differs between groups, and this is a contrast you cannot verify with the EMMEANS command (at least I did not find an easy way).

Now, in models with many factors it is a bit tricky to write down the /TEST line, the sequence of 1/2, 1/4 etc, called L matrix. Typically if you get the error message: "the L matrix is not estimable", you are forgetting some elements. One link that explains the receipt is this one: https://stats.idre.ucla.edu/spss/faq/how-can-i-test-contrasts-and-interaction-contrasts-in-a-mixed-model/

  • $\begingroup$ What a great answer. You can make it even better if you 1. summarise the content of your link and 2. explain what you are doing statistically $\endgroup$
    – Ferdi
    Dec 7, 2017 at 13:29

I don't know SPSS syntax particularly well, but, if I understand your situation correctly, the significant interaction means that, in order to adequately assess the significance of your main effects, you'll need to do separate analyses. I think the best way to proceed is to do separate repeated measure analyses for each level in your grouping factor. Perhaps someone else can speak better to the question of how to handle correcting for multiple comparisons during post-hoc analysis, but I'm pretty sure you still need to use a correction. You might try Tukey's, as a multiple comparison correction!

  • $\begingroup$ Thank you for your answer. If I understood correctly, you suggest the solution 1), to conduct two separate repeated measures ANOVAs, one for each group, with time as the within subjects indipendent variable (3 levels) and then, if significant, compare the main effects with Tukey's correction (or Bonferroni, I guess, isn't it ok?). Have I understood correctly? $\endgroup$
    – Federico
    Nov 18, 2012 at 11:55
  • $\begingroup$ In this case, using SPSS I selected "Data/Split file..." and entered the grouping variable. Is this correct? This way, I found a slightly significant ANOVA (p = 0.044) for the control group, but Bonferroni (it doensn't allow me to do Tukey) comparisons are all non significant... How should explain this? Is the ANOVA result a I type error? $\endgroup$
    – Federico
    Nov 18, 2012 at 11:57

In short. There is no globally accepted convention for these situations. Some will use Bonferroni corrections. Some will force the Tukey HSD framework to dance for them (e.g. Maxwell & Delaney). In contrast...

COMPARE(time) ADJ(BONFERRONI)", just after "/EMMEANS=TABLES(newgroup*time)

... does seem to use the Bonferroni correction. However, this approach is likely conservative, especially in the face of Holm-Sidak style corrections. (ESPECIALLY if you don't use the MSW as the error term for your post-hoc comparisons).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.