Two-Way ANCOVA: It is necessary to include a non-significant interaction term in the model?

Question

The objective is to test whether the adjusted group means are equal. The question is whether ANCOVA requires the interaction term, for this objective.

In all the guides I am reading, after testing for assumptions, you run a two-way ANCOVA (with one covariate) using this formula:

Model <- Anova(aov(DependentVar ~ Factor1 * Factor2 + Covar1, 
           data = Dataset), type = "III")

If the interaction term is not significant (Factor1:Factor2 ; p = 0.65), which is actually already known here, is it appropriate to re-run this as:

Model <- Anova(aov(DependentVar ~ Factor1 + Factor2 + Covar1, 
           data = Dataset), type = "III)

In this case, including the non-significant (p = 0.65) interaction term drops very significant main effects out of significance.

Answer 1

It looks like other responses have already addressed the fact that there is no absolute rule that an interaction needs to be included. I'll just echo briefly that the decision of including an interaction should be driven by theory, and I'd like to use my answer to just fill in some context about why that matters.

First, consider what you are analyzing when you include only main effects for two different factors. The primary advantage of a two-way AN(C)OVA, or really any factorial ANOVA for that matter, is that you can look at the interaction between two or more factors. If all you end up wanting to look at are the main effects of the factors and not how they interact, then all you really want is two separate one-way ANOVAs. The only advantage of doing a two-way ANOVA with no interaction versus two one-way ANOVAs is that you don't have to worry about adjusting the p-value for multiple observations if you use a two-way.

Second, it's useful to think about what the aim of developing statistical models is. I highly recommend Dr. McElreath's book Statistical Rethinking as a reference for how to think about what our models really mean in the real world. In short, a statistical model is always an approximation to the real world and thus always has some error because we are making assumptions to simplify the problem. Since we always have error and our models are never right, we need to think about what information from a model is actually useful for us. In this case, use is relative. Unfortunately, many people associate statistical significance with utility. The result is that models often get built using a method that McElreath calls "star gazing" where essentially we run a model and then only keep the variables that are significant (e.g., have the *, **, *** indicators of statistical significance). This rarely produces a useful model; instead, learning about what variables do emerge as significant is really only useful when we contextualize those findings with our theories and expertise. So, in this case, choosing not to model an interaction because it is not significant is not a particularly strong argument for that modeling decision, unless there is some other reason that we would expect that the interaction is not relevant (in which case the absence of a significant effect would be some evidence that our a priori theory).

Finally, there's a fairly serious ethical/scientific rigor issue of repeatedly running a model and dropping or adding variables based on statistical significance. As a general rule, making a decision to adjust a model based only on the statistical significance of the results can lead to p-hacking. Essentially, it's possible to manipulate data and models to produce significant results even when there is no true effect or relationship. Each time we run a statistical test/build a model, we are accepting some level of random chance that we spuriously detect something that is not a true result (this is whatever we select our $\alpha$ to be, which is usually 0.05). As a result, every new model we try is increasing the chances that we stumble across a significant result and make a Type I error (rejecting the null when the null is actually true). Where this becomes a real problem is when we choose to drop non-significant results. In the case of ANOVA or really any general linear model, the aim is to separate sources of covariance among variables and variance within variables to parse out what effects there are. Non-significant variables usually account for at least some of this covariance (even if it is a really small amount), so removing those variables allows other variables still included in the model to account for potentially more of that covariance and thus have larger potential effects. These kinds of model manipulations are essentially double dipping your data (i.e., using the data to fit a model and then using the results of that model to fit a "better" model).

So, in short, there is no hard rule that you must include an interaction; instead, this is a decision that you should evaluate for your data, research question, and research aims. You should also take into account the potential implications of making model changes based only on the results of null-hypothesis tests. There's no right answer per se, but it's important that you as a researcher/data scientist are balancing these kinds of decisions

Answer 2

There are two different definitions or understandings of the term ANCOVA.

The first and a broader one is "Any linear model containing continuous/scale predictors besides factors (categorical predictors). The continuous predictor then receives an argot name "covariate". Often it this broader sense "covariate" is just a quantitative independent variable in any regression, not only linear model.

The second and narrow one is a special case of the former, often spelled in full as "the analysis-of-covariance model". It is the linear model for inference, with categorical factor(s) and quantitative covariate(s), where factor-covariate interaction (Fac*Cov) is nonsignificant and on this ground may therefore be dropped, reducing the full model to the Y = const + Fac + Cov. The nonsignificance of the interaction corresponds to the "homogeneity of regression slopes" assumption. The aim of such ANCOVA is double: (i) reduce SSerror and likely make the factor(s) more significant through this; (ii) if the factor and the covariate correlate, that is, factor levels differ by the mean value of the covariate - then partial out the effect of it from the factor's effect, thus purifying the latter, i.e. "adjust factor for covariate" - remove the means' shift. If homogeneity-of-regression-slopes assumption is not met, it would be difficult to interpret the factor's effect not statistically, but conceptually in many real life domains.

Irrespective of broad or narrow understanding of ANCOVA, factor by factor interaction is what has no relation to the definition of the term. You may built models with or without such interaction, and it all be ancovas.

See also.

Answer 3

There is nothing in statistical theory or practice which requires you to include any interaction, or any main effect for that matter. You include in your model the variables which your scientific theory has suggested and you include any interaction which that theory has suggested. You would then present that model to the reader. If you now decide to modify the model in the light of the data you need to make it clear to the reader what you have done. If you do not you risk mis-leading them.

Having said all that it does seem very unusual that removing a negligible interaction has a massive effect on the whole model but in the absence of any further information we cannot speculate on how that has come to pass.

Answer 4

When using frequentist methods, it is invalid to remove the interaction from a linear model just because of a hypothesis test, because that invalidates the estimate of residual variance, i.e., it results in an underestimation of $\sigma^2$. This is a form of model uncertainty discussed at more length in RMS.

In a confirmatory randomized experiment it is commonplace to have a "no interaction" model for the primary analysis and a pre-specified secondary analysis using an interaction model. The most cohesive approach IMHO is a Bayesian analysis in which the prior distribution for the interaction effect is elicited before data analysis. This allows interactions to be "half in" and "half out" of the model.