Interpreting Two-way repeated measures ANOVA results: Post-hoc tests allowed without significant interaction?

Question

I have conducted a study where I employed a two-way repeated measures design to investigate whether subjects respond to a main treatment effect. What I'm mainly interested in is whether subjects subsequently compensate for the treatment effect by returning to a control level, or whether the initial differences persist for the duration of the treatment (time x treatment interaction).

My factors:

Treatment: Between-subjects factor (3 levels)

Exposure Time: Within-subjects repeated measure (4 levels - pre-exposure values not included)

Treatment x Time Interaction:

Error bars = Standard deviation

My ANOVA results: Sphericity was taken into account: p = 0.25. Adjusted values told the same story.

Both main effects, 'Treatment' and 'Time' were significant. Treatment effect had a large effect size (0.4). (sensu Cohen) Time, isn't really that interesting on its own, had a small effect size (0.08) The interaction was not significant (p = 0.3) and had a tiny effect size (0.04).

My problem: The graph appears to show that one of the treatment levels interacts with exposure time (fig. d). (ie the lines are not all parallel from day 4 - 14) The ANOVA results however suggest that that the difference between treatments persists (the lines remain parallel). I have looked at pairwise comparisons within levels of the main effect.

My question: I am tempted to included post-hoc comparisons to compare treatment levels at each time interval to show that the difference between groups disappears on day 11. Does this defeat the purpose of running an ANOVA, which clearly tells me that I should just state that there was a main effect (ie the treatment groups remained different in fig d). I therefore shouldn't really fish around for significant differences.

The literature that I'm comparing my work to employs ANOVA, but then also puts pairwise comparisons all over the place. I'm a fan of effect sizes, rather than p-values, so feel justified to just highlight the main treatment effect.

My main conclusion could be very different depending on whether I go with the ANOVA (ie no interaction -> no recovery) or whether I include pairwise comparisons at each time point (partial recovery).

Side note: Mixed effects models in lme gave the same result. I stuck with ANOVA as my audience is more likely to relate to it.

Thanks for the feedback.

Answer 1

Well, this seems like an old problem, but I'm still tempted to share my two cents.

I am tempted to included post-hoc comparisons to compare treatment levels at each time interval to show that the difference between groups disappears on day 11

I don't think you should do this. After all, ANOVA said that the interactions are not significant. The only thing I see in plot (d) is that on day 0, there does not seem to be a difference between Treatment and Control. But you probably already knew this. Just report the main effects, and file the rest under "Speculations".