Which is the correct way to deal with insignificant main effect of Group condition? Stick with interaction effect in RM-Anova or perform ANCOVA?

Question

I wanted to ask conceptual about what to do with main effects?

Assume, I have randomly assigned equal two groups(let's say CBT, Control; N1=25; N2:25). I collected the depression levels at three time points (pre, pos and follow-up). At pre level, using independent sample t-test, groups did not differ each other significantly)(p>0.05).

Now using Two-Way Repeated Measures ANOVA(one-between, one within), I found the time main effect is significant(p<0.05) while group main effect is not significant(p>0.05). However, I found the interaction is also significant(p<0.05). Now obvious logical procedure so far, I need to look for EMM post-hoc comparisons at which time points and groups differ. So, I performed the comparison tests for the interaction with adjusting Bonferroni:

Before ending the results let's assume my results look like this:

pre-condition:

for control: 61(SD:18) for treatment: 60(SD:19)

post-condition:

for control: 59 (SD:17) for treatment: 45 (SD:18)

follow-up:

for control: 56 (SD:19) for treatment: 40 (SD:16)

I found between group differences which is significant at post and follow up level.

I also found pre to post level, treatment condition is significantly lower while control group not. Similar result is also valid for pre to follow-up level but not for post to follow up.

When it comes to my conceptual question, my colleague warned me not to look pairwise comparisons unless I found both main effect is significant. Because treatment over control across time cannot be interpret unless you have a main effect of condition. Bottom line is my logic tells me I did the correct way to interpret the results, but I cannot find the exact way or source to defend my method especially two-way repeated measures ANOVA. Some of my colleagues also pointed out if you want to not be affected by pre level condition to assess main effect of group condition, use ANCOVA instead since it was better choice when comparing between groups. Because ANCOVA is an alternative approach to handling data when pre-post test designs especially groups formed by random assignment. And also they suggesting this strategy sourcing this book: ANOVA:Repeated Measures by Girden, E.(1992).

In short, how can I interpret this insignificant group main effect? Or is this really critical step to interpret regarding my study design and results whether to interpret interaction?

I've read related questions and comments but still not sure to know that whether I need to interpret the insignificant main effect. Or even whether I need to perform ANCOVA for adjusting pre-level at this step instead of RM-ANOVA?

Answer 1

Interaction effects that are significant when the main effect is not is always a bit challenging to unpack. In brief, this means that, on average, there is no overall difference between treatment groups. However, if you were to look at the time trend for each group (from pre to post to follow-up), the trend will not be consistent for the two treatment groups.

One example might be where the groups start out the same, then one treatment condition does better at post and they flip at follow-up. This is just one possible example...but the idea is that something changes in the progression of time for each comparison via treatment.

Now, the interesting part is that RM-ANOVA is higher powered than your basic t-test. So, this means that you might do a t-test comparison at each time (comparing treatments), and none of the tests come up statistically significant. But, the interaction might be significant in the RM-ANOVA. While it will be difficult to tease apart exactly where the group difference might be, the RM-ANOVA is probably the better result to report.

My suggestion would be to generate a means-plot for each treatment group for the different time points. This may be more informative at revealing the trending change over time for the two groups.