Batch effect significance

Question

I'm doing a project where i study bending angles in plants, with different genotypes and treatments. I have done the experiment 4 times, so i have 4 different batches. I'm trying to take into account the batch effect in my analysis. When doing a three-way ANOVA with interaction in R (model formula :

aov(Angle~Genotype*Treatment*Batch)).

The ANOVA results tells me that i have a statistically significant batch effect, but i don't think that this effect is biologically significant (one batch differs from the other 3, with a difference in bending angle of about 5°). As I'm studying the interaction in my data, i also get results for the interaction between Genotype:Treatment:Batch, which is not significant (p =0.0688, which is agree is not statistically really strong). My analysis mainly focuses on the Genotype:Treatment interaction. If i'm not mistaken, the fact that the three factor interaction is not significant means that the Genotype:Treatment interaction is not different across all levels of the batch factor.

Does that mean that the batch factor is not relevant in my analysis ?

Thank you for reading me :)

Answer 1

First, unless you have a completely balanced design without missing data you should not be using aov() to evaluate the model. From the help page:

aov is designed for balanced designs, and the results can be hard to interpret without balance: beware that missing values in the response(s) will likely lose the balance.

See this page for further discussion. Seting up a standard lm() model and applying the default Type II ANOVA provided by the Anova() function in the R car package is a better approach if your design is unbalanced.

Second, you need to evaluate both statistical significance and practical significance whether it seems to work in your favor or not. You say that a "statistically significant" bending-angle difference of 5° isn't "biologically significant." On the other hand, although a 3-way interaction-coefficient p-value of 0.0688 doesn't pass the usual arbitrary p < 0.05 cutoff for "statistical significance," the magnitude of the interaction might be "biologically significant."

Third, even if the 3-way interaction isn't "significant" some batch effect might be. That says nothing about 2-way interactions of Batch with either of Treatment or Genotype, or the individual coefficient for Batch that estimates the linear association of Batch with outcome. Even if a bending angle of 5° is small in comparison against the treatment and genotype effects, ignoring it completely might lower your power to detect real effects of primary interest.

Fourth, you need to apply your understanding of the subject matter to decide how to correct for batch effects. Sometimes it's OK just to include a linear term for Batch, with the advantage of fewer coefficients to estimate than for the fully crossed interaction model:

myModel <-lm(Angle ~ Treatment*Genotype + Batch)

That corrects for systematic differences in baseline values of Angle among batches while assuming that the extra associations of Treatment and Genotype with outcome are independent of Batch. Is that assumption reasonable? You should make such decisions based on your understanding of the subject matter, ideally before you run the full model.