I have difficulty interpreting ANOVAs results

Question

I have a dataset of genotype performance (that's my independent variable) in different growing environments. I want to know if there are interactions between genotypes and the different environments and if the genotypes can be distinguished from each other regardless of the environment. I'm using statsmodels in python with the functipon anova.anova_lm

The results of the ANOVA (1) with genotype*environment interaction are as follows:

             sum_sq       df     F          PR(>F)
genotype     0.408288   103.0    3.334366   3.181776e-15
ENV          2.170352     8.0  228.204205   8.152433e-175
genotype:ENV 0.654210   824.0    0.667842   1.000000e+00
Residual     4.277378  3598.0   NaN          NaN

This is interpreted as the genotype and environment contributions being significant in the response variable, but the genotype and environment interaction with a p-value of 1 is not significant.

However, if I run another ANOVA (2) using only genotypes as a variable, I get the following result:

           sum_sq     df     F        PR(>F)
genotype   0.167824  103.0  0.8209   0.905164
Residual   8.683694  4375.0  NaN      NaN

The p-value of 0.9 indicates that there are no significant differences between different genotypes. Why do genotypes not contribute to the response variable here but do in ANOVA 1?

Finally, if I run another ANOVA (3) considering only the interaction between genotype and environment, but not their individual terms, I get:

              sum_sq     df     F       PR(>F)
genotype:ENV  4.860024  935.0  4.372   3.325465e-215
Residual      4.277378  3598.0 NaN     NaN

The interpretation is that there are significant differences between genotype and environment interactions, which seems to contradict the results obtained in ANOVA 1.

All I want to answer is whether there are significant differences in the response of different genotypes.

Answer 1

Under the traditional backward selection procedure, you should try the models in this order:

1. fit $y$ related to genotype + ENV + genotype:ENV (ANOVA-1)
2. drop the genotype:ENV interaction because it is not significant
3. fit $y$ related to geneotype + ENV (this was not shown)
4. fit $y$ related to intercept + genotype (ANOVA-2)
5. genotype is not significant on its own (but might have been in the presence of ENV if it was tested).

I guess that, in your case, ENV would have been significant in step 3, and possibly genotype. You'll have to show us.

Why do genotypes not contribute to the response variable here but do in ANOVA 1?

We can't tell the answer from what you gave in the question, but your analysis of ANOVA-1 is incorrect. You don't know if the genotype main effect is significant in ANOVA-1 because you should only evaluate the interaction in ANOVA-1 under the normal process of backward selection. You have to move to step 3 to then evaluate the main effect.

ANOVA-3 does not contradict ANOVA-1

In ANOVA-3 you only tested the interaction which was significant. In ANOVA-1, the model says that the interaction is not significant when in the presence of the main effects. These are completely different statements and do not contradict.

Answer 2

First, it looks like the default in statsmodels.stats.anova.anova_lm( ) is to perform "Type I" ANOVA, as it is with the base anova() function in R.

Those results can be misleading unless the design is balanced. See this page and its links. That's true even with a simple additive model without an interaction term.

In your situation there probably isn't a big problem with that, as both the genotype and ENV associations with outcome are strong in the full model with the interaction, ANOVA(1). But that can be a big hidden "gotcha" in other circumstances.

With respect to model ANOVA(2), you are evidently suffering from omitted-variable bias. Leaving out one predictor that's associated with outcome can bias the estimates for the remaining predictor. It can even go so far as in your situation, where the remaining predictor appears to be "insignificant."

With respect to model ANOVA(3), it can be hard to interpret interaction terms when you omit the corresponding individual predictors. See this answer for example.