I wanted to ask conceptual about what to do with main effects?
Assume, I have randomly assigned equal two groups(let's say CBT, Control; N1=25; N2:25). I collected the depression levels at three time points (pre, pos and follow-up). At pre level, using independent sample t-test, groups did not differ each other significantly)(p>0.05).
Now using Two-Way Repeated Measures ANOVA(one-between, one within), I found the time main effect is significant(p<0.05) while group main effect is not significant(p>0.05). However, I found the interaction is also significant(p<0.05). Now obvious logical procedure so far, I need to look for EMM post-hoc comparisons at which time points and groups differ. So, I performed the comparison tests for the interaction with adjusting Bonferroni:
Before ending the results let's assume my results look like this:
for control: 61(SD:18) for treatment: 60(SD:19)
for control: 59 (SD:17) for treatment: 45 (SD:18)
for control: 56 (SD:19) for treatment: 40 (SD:16)
I found between group differences which is significant at post and follow up level.
I also found pre to post level, treatment condition is significantly lower while control group not. Similar result is also valid for pre to follow-up level but not for post to follow up.
When it comes to my conceptual question, my colleague warned me not to look pairwise comparisons unless I found both main effect is significant. Because treatment over control across time cannot be interpret unless you have a main effect of condition. Bottom line is my logic tells me I did the correct way to interpret the results, but I cannot find the exact way or source to defend my method especially two-way repeated measures ANOVA. Some of my colleagues also pointed out if you want to not be affected by pre level condition to assess main effect of group condition, use ANCOVA instead since it was better choice when comparing between groups. Because ANCOVA is an alternative approach to handling data when pre-post test designs especially groups formed by random assignment. And also they suggesting this strategy sourcing this book: ANOVA:Repeated Measures by Girden, E.(1992).
In short, how can I interpret this insignificant group main effect? Or is this really critical step to interpret regarding my study design and results whether to interpret interaction?
I've read related questions and comments but still not sure to know that whether I need to interpret the insignificant main effect. Or even whether I need to perform ANCOVA for adjusting pre-level at this step instead of RM-ANOVA?