How does the interpretation of main effects in a Two-Way ANOVA change depending on whether the interaction effect is significant?

Question

As far as I can tell this question has never been asked before. There are several questions that touch on related issues, but as far as I can see none of them have provided a definitive answer to this question. Furthermore, in at least one case the most upvoted and second most upvoted answer have implicitly disagreed with each other.

Textbooks often caution against interpreting main effects in Two-Way ANOVA when the interaction effect is significant. Here's a (mild) example of this, from pp562-563 of Hatcher's "Step-by-step Basic Statistics Using SAS: Student Guide":

When you perform a two-way ANOVA, it is possible that you will find that (a) the interaction term is statistically significant, and (b) one or both of the main effects are also statistically significant. When you prepare a report summarizing the results, you will certainly discuss nature of your significant interaction. But is it also acceptable to discuss and interpret the main effects that were significant?

There is some disagreement between statisticians in answering this question. Some statisticians argue that, if the interaction is significant, you should not interpret the main effects at all, even if they are significant. Others take a less extreme approach. They say that it is acceptable to interpret significant main effects, as long as long as the primary interpretation of the results focuses on the interaction (if it is significant).

Do I assume correctly that the hard-line stance ("you should not interpret the main effects at all") is incorrect, but that the interpretation must nonetheless change if the interaction effect is significant? If so, how does the interpretation change?

Answer 1

There is less to this issue than it seems. The real answer isn't that you cannot interpret the main effects at all, but rather that it is very difficult to interpret them correctly. The reason for the warning not to interpret the main effects is because people will inevitably interpret them incorrectly.

If there isn't an interaction term included in the model, the main effects have a straightforward meaning: is there variation amongst the levels of the factor in question? If there is an interaction in the model, the main effects don't mean that. In fact their meaning is hard to convey and it depends on how the model was fit and how it was tested. In the abstract, I cannot tell you exactly what they mean in any given model. However, interpreting them as you would if there weren't an interaction would be incorrect. What is important for this issue is not whether or not the interaction is significant, but whether or not the interaction was included in the model in the first place.

If the interaction is sufficiently non-significant for your purposes, and you want to test and interpret the main effects, the simplest thing to do would be to drop the interaction and re-fit / re-test the model. Note that this procedure, if not a-priori, comes with all the usual caveats about fishing and threats to the validity of the hypothesis tests.

Answer 2

This is an interesting question. Since I don't like gross generalization, I am going to disagree with the suggestion that you should "never" interpret the main effects at all if an interaction is present. Never is just to strong (even if some might argue that there are clear situations where the interaction tells you what you need to know. To do this, I will provide a counter example and a link to a paper in the Journal of Consumer Psychology where this precise question was asked & then answered by the editor.

In the paper, the following question is asked: Can I make conclusions based on an interaction being significant without testing the simple or main effects?

Here is the hypothetical situation that is given:

"A researcher predicts a crossover interaction between two variables (one continuous, the other dichotomous) that is tested via a linear contrast regression model containing the appropriate main effect and interaction terms. The interaction term receives a significant coefficient. The researcher then concludes that the data support the predicted interaction. Is this an appropriate conclusion? More specifically, given that a significant interaction could occur for data patterns that differ from the one predicted (e.g., a non crossover pattern, a crossover pattern in the opposite direction), is it not necessary to undertake the appropriate simple main effects tests to establish whether the data actually support the differences predicted by the crossover interaction?"

The argument / response that is provided by the editor in favor of always reporting that main effect even when the interaction is present (demonstrating a counter example):

"Yes, it is imperative to defend a statement purported to describe data with a statistic. The scenario you have described for regression would be like obtaining a significant F test for the A×B interaction in ANOVA and then doing no further investigations — that is, both the plot of the cell means and the tests of simple effects to substantiate claims about precisely which means are significantly different from each other. You are absolutely right that a mere significant regression coefficient associated with an interaction term yields no detailed information about the nature of that interaction. If you see someone trying to do this—make a claim about their data without a statistic to support that claim (regardless of whether in regression or ANOVA or anything else for that matter), nail them."

While others may not, I agree with the argument proposed by the author and find the counter-example compelling enough to reject the hardline claim you described that one should not interpret the main effects if the interaction effects are significant.

Answer 3

When you have an interaction, the real issue is whether the main effects (significant or not) are “descriptive” or “misleading”.

Here is an example of data with a significant interaction and “descriptive” main effect for Task Presentation (Computer does better than Paper overall, and for Easy Tasks and for Hard tasks)

Here are two examples of data with a significant interaction and “misleading” main effect for Task Presentation (computer does better than paper overall, but not for both Easy and Hard tasks)

and here is an example of a null main effect that is “misleading” because although there is no main effect of Task Presentation, there is a Task Presentation effect for both Easy and Hard tasks.

Kind of a long answer, but it is an important issue.

Here is the PowerPoint file that contains these examples, and the course.