1
$\begingroup$

I wanted to ask conceptual about what to do with main effects?

Assume, I have randomly assigned equal two groups(let's say CBT, Control; N1=25; N2:25). I collected the depression levels at three time points (pre, pos and follow-up). At pre level, using independent sample t-test, groups did not differ each other significantly)(p>0.05).

Now using Two-Way Repeated Measures ANOVA(one-between, one within), I found the time main effect is significant(p<0.05) while group main effect is not significant(p>0.05). However, I found the interaction is also significant(p<0.05). Now obvious logical procedure so far, I need to look for EMM post-hoc comparisons at which time points and groups differ. So, I performed the comparison tests for the interaction with adjusting Bonferroni:

Before ending the results let's assume my results look like this:

pre-condition:

for control: 61(SD:18) for treatment: 60(SD:19)

post-condition:

for control: 59 (SD:17) for treatment: 45 (SD:18)

follow-up:

for control: 56 (SD:19) for treatment: 40 (SD:16)

I found between group differences which is significant at post and follow up level.

I also found pre to post level, treatment condition is significantly lower while control group not. Similar result is also valid for pre to follow-up level but not for post to follow up.

When it comes to my conceptual question, my colleague warned me not to look pairwise comparisons unless I found both main effect is significant. Because treatment over control across time cannot be interpret unless you have a main effect of condition. Bottom line is my logic tells me I did the correct way to interpret the results, but I cannot find the exact way or source to defend my method especially two-way repeated measures ANOVA. Some of my colleagues also pointed out if you want to not be affected by pre level condition to assess main effect of group condition, use ANCOVA instead since it was better choice when comparing between groups. Because ANCOVA is an alternative approach to handling data when pre-post test designs especially groups formed by random assignment. And also they suggesting this strategy sourcing this book: ANOVA:Repeated Measures by Girden, E.(1992).

In short, how can I interpret this insignificant group main effect? Or is this really critical step to interpret regarding my study design and results whether to interpret interaction?

I've read related questions and comments but still not sure to know that whether I need to interpret the insignificant main effect. Or even whether I need to perform ANCOVA for adjusting pre-level at this step instead of RM-ANOVA?

$\endgroup$

1 Answer 1

0
$\begingroup$

Interaction effects that are significant when the main effect is not is always a bit challenging to unpack. In brief, this means that, on average, there is no overall difference between treatment groups. However, if you were to look at the time trend for each group (from pre to post to follow-up), the trend will not be consistent for the two treatment groups.

One example might be where the groups start out the same, then one treatment condition does better at post and they flip at follow-up. This is just one possible example...but the idea is that something changes in the progression of time for each comparison via treatment.

Now, the interesting part is that RM-ANOVA is higher powered than your basic t-test. So, this means that you might do a t-test comparison at each time (comparing treatments), and none of the tests come up statistically significant. But, the interaction might be significant in the RM-ANOVA. While it will be difficult to tease apart exactly where the group difference might be, the RM-ANOVA is probably the better result to report.

My suggestion would be to generate a means-plot for each treatment group for the different time points. This may be more informative at revealing the trending change over time for the two groups.

$\endgroup$
6
  • $\begingroup$ What I've really understand from your saying, is that RM-ANOVA is a correct strategy to reveal between group effect over time. But I don't understand you're saying that I shouldn't look for EMM post-hoc comparisons at which time point and groups differ? In short, state descriptive statistics and report this interaction effect with plots to interpret. But this is really confusing me as if I did only the reports half of it. $\endgroup$
    – Connor
    Commented Jun 1, 2023 at 14:57
  • $\begingroup$ I also did perform repeated measures ANCOVA by the way. And I've found main effect of group is significant; however time or interaction is not significant. Group differences revealed at the post and follow-up adjusting pre-condition levels of depression. I'm bit confuse right now. When I perform RM-ANOVA interaction and time is sig; but if i perform RM-ANCOVA only group main effect is sig. Since I don't know really which to use (although my colleague strongly recommend that I should perform RM-ANCOVA because most RCTs uses ANCOVA to adjust pre level of the DV), when i study RCT design. $\endgroup$
    – Connor
    Commented Jun 1, 2023 at 15:04
  • $\begingroup$ The lack of significance for the RM-ANCOVA (which may not be the best label here) is because time is treated as categorical in the RM-ANOVA, but as a scalar covariate in the ANCOVA model. Essentially, these are indeed two different models (so the results would not be expected to be the same). $\endgroup$
    – Gregg H
    Commented Jun 1, 2023 at 15:50
  • $\begingroup$ Thanks for quick and straight answer. With regard to my questions, you say RM-ANOVA is a correct strategy. But knowing that most RCTs uses RM-ANCOVA, what should I do perform? (Considering I want to look for whether my treatment groups superior compared to control in post or follow-up level) Also, I should not perform EMM post-hoc comparisons in RM-ANOVA, instead I should just indicate descriptive stats and interaction plot. Am I correct for RM-ANOVA based on your answer? $\endgroup$
    – Connor
    Commented Jun 1, 2023 at 16:17
  • $\begingroup$ In the literature, as here: researchgate.net/publication/… says ANCOVA has the highest power; and ANOVA is the least power and is the worst method with no evidence of an intervention effect even when the treatment by time interaction is statistically significant. In contrast to this, ibm.com/support/pages/… I found using ANCOVA is a wrong way to deal with in here. $\endgroup$
    – Connor
    Commented Jun 1, 2023 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.