Repeated measures ANOVA with significant interaction effect, but non-significant main effect

Question

I am running a two-way repeated measures ANOVA (main effects: Time, Condition). The result is that the main effect of time is significant (P<0.05), the main effect of condition is not significant (P>0.05), and the interaction effect (time*condition) is significant (P<0.05).

Considering there is a significant interaction effect, we have ran Tukey post hoc testing to decompose the data points at each time and determine if differences exist. Although to my understanding this is acceptable, our approach has recently been questioned as an individual has suggested you need all main effects to be significant prior to further investigation into the significant interaction effect.

Does anyone have any thoughts/articles that may support/refute my approach. Kind regards,

Answer 1

Inadvertently, you're comparing apples and oranges. Once an interaction is included in the model, the interpretation of the main effect changes. With the interaction, the main effect of "condition" is the expected difference at time 0, and the effect of time is the slope of time trend for the referent condition.

The most likely scenario explaining your finding here is that the interaction projects the difference and slope of these effects to 0 at the specific timepoint. It of course does not mean you should remove them from the model - the interaction effect will no longer be interpretable. In the example plot below, I show two trendlines for a continuous variable X for a binary factor W. When the interaction is included in the model, the X trend is significant but the W effect is no longer significant. That's because when X=0, the values of Y are exactly the same whether W=0 or 1. However, it's clear from the graphic that both W and X are "significant" variables in the sense that the response is strongly predicted by their inclusion in the model - specifically the interaction.

Answer 2

If you have a significant interaction and you are using typical linear modeling, then the "main effects" represented by individual-predictor coefficients are at best hard to interpret and at worst outright confusing.

In the usual coding of predictors in R, what is typically taken to be the "main effect" for a categorical predictor like condition is its regression coefficient in a linear model. With an interaction in your situation, that is its association with outcome only when time = 0. That might be too early to see an association of condition with outcome if the conditions only differed starting at time = 0. See the plot in the answer from @AdamO, treating x as your time and w as your treatment.

The corresponding "main effect" for a continuous predictor like time would be its association with outcome when condition is at its reference level. If that reference level is "control" it's quite possible that you wouldn't have a "significant" "main effect" coefficient for time either, if the outcome is constant without an intervention.

The interaction coefficient tells you how much the association of time with outcome depends on the value of condition, and vice-versa. If that's appreciable (not even necessarily "significant"), the apparent "main effect" of one predictor can depend on the coding of a different predictor with which it interacts.

The best way to evaluate the importance of a predictor involved in an interaction requires evaluating all of the terms involving it, for example with likelihood-ratio tests or Wald tests on its sets of coefficients.

Finally, there's no requirement to do an initial ANOVA if you already planned to do all of the pairwise comparisons. So your Tukey pairwise comparisons are OK to perform in any case.

Answer 3

The motivation for not looking for interaction effects if there are no significant main effects is that one might end up comparing too many different models and could end up with the multiple comparisons problem. The scheme to only analyse interaction effects after main effects have been found is a way to reduce the probability for a false positive.

However, you can very well have a significant interaction without one or both main effects being significant.

Often, you have that the main effect correlates with the interaction effect and when the interaction is significant then the main effect will also be likely significant when we fit the model without the interaction, but this does not need to be the case. Here is an extreme example

Let $$z = x \cdot y + \epsilon$$ with $x,y,\epsilon \sim N(0,1)$ This looks below

The variable $z$ when plotted versus $x$ or plotted versus $y$ is on average equal to $0$ independent from the value of $x$ or the value of $y$, and you will not find significant main effects.

But as a function of 'both' $x$ and $y$ and more specifically the product $x\cdot y$ you have a strong and clear effect.

In such a case it would be nonsense to ignore the interaction because there are no main effects.

Related question: One of the main effects not significant, but interaction term significant

Answer 4

I am late to the party here, and there are already good answers. But I don't think the sentence:

an individual has suggested you need all main effects to be significant prior to further investigation into the significant interaction effect.

has been fully addressed (although it has been part of several answers, especially that of Sextus).

Sextus notes that exploring all interactions can lead to a lot of comparisons. That is certainly true.

However, I think the decision of whether to look at interactions, and which ones to look at, should be made before you run any other analyses and, ideally, before you gather data. Also, even if you decide, a priori to base it on some results, it should not be based on whether any effects are significant, but on the effect size.

It is possible to have both main effects be very close to 0 and yet have an interaction. This will happen, for instance, when a treatment has opposite effects in two groups.