If an ANOVA indicates no main effect and no interaction, should the lack of interaction be stated?

Question

I measured response variable $Y$ at three levels of factor $A$ and four levels of factor $B$, $n=6$ reps/treatment. Results include

• $A$ has strong effects on $Y$.
• $B$ has no effect on $Y$
• There is no $A*B$ interaction

I would like to report all three results (the second two are actually more interesting than the first since all were expected). So far I have:

"The effect of A on Y was significant (ANOVA, $P<0.001$) and $Y$ was different at each level of $A$ (Tukey HSD, $P<0.001$). There was no effect of $B$ on $Y$, and there was no interaction between $A$ and $B.$ This implies that $B$ had no effect on $Y$ and that the effect of $B$ on $Y$ was similar for all levels of $A$."

(Text with bold emphasis is in question)

This approach minimizes the double negative as in "the effect of $B$ on $Y$ was not different at different levels of $A$". Still, neither sounds 'right' to me, thus my post.

In my field, as in many fields of science, non-significant responses are rarely mentioned in the text, and even more rarely are they interpreted, but in this case, previous studies find that $B$ almost always affects $Y$ and the lack of an interaction is notable.

Answer 1

Well, it depends if the interaction was your main hypothesis or not. If this the case, then you are encouraged to report the negative result, otherwise you can simply refit your model (without the B and A:B terms) to get a better estimate of A.

Now, the part of your conclusion that you emphasized doesn't sound correct to me. You can only prove that an observed difference of means is different from 0 (or any other fixed value, according to your alternative hypothesis), you cannot "accept" the null. If your test is non-significant, it simply means that you cannot reject $H_0$. Non-significant results can also reflect lack of power (Type II error).

Also, rather than simply reporting crude p-values, it would be better (and it actually follows the APA recommendations) to also report some kind of effect size or difference of means, together with your inferential results.

Here is an example for reporting results from a factorial ANOVA (it has to be rework to fit your specific experimental design since your factors have a lot of levels):

A two-way analysis of variance yielded a main effect for A factor, $F(\nu_1,\nu_2) = 0.00$, $p < .05$, such that the average "value" was significantly higher in the $a_1$ condition (Mean=0.00, SD=0.00) compared to $a_2$ (Mean=0.00, SD=0.00) and $a_3$ (Mean=0.00, SD=0.00, Tukey HSD, all p<0.05). The main effect of B was non-significant ($F(\nu_1,\nu_2) = 0.00$, $p = 0.00$), and no interaction effect was found significant ($F(\nu_1,\nu_2) = 0.00$, $p = 0.00$) indicating that the effect of A on Y was not significantly different across the B levels.

Answer 2

Please let me know if you have replicates in your experiment. As you mention using Tukey HSD, I am guessing you don't have any replicates. If you experiment analyzes test of additivity in a two-way factorial Analysis of Variance (ANOVA) with one observation per cell, then please read ahead or otherwise ignore my solution.

Assuming I am thinking in the right direction, it may be possible that B affects A, but it doesn't affect Y in form of a multiplicative term like A*B. In general, B can affect Y in even if the interaction A * B is not significant.

The reason for this is the assumption of special form of relation between A and B (ie A*B) that affects Y in Tukey HSD Test. Please see the wikipedia article about Tukey HSD for further details.

I am sorry if this was not the test that you were conducting.

But even if the interaction is not statistically significant, it doesn't imply that B doesn't affect Y. Even if this is a randomized experiment and you are allowed to relate significance and causation, but not able to reject the null hypothesis $H_o$ (ie p-value is $\geq \alpha$ ) doesn't mean that you accept $H_o$.

Please accept my apologies, if I completely misunderstood you.

---

Update after David's details

A few common themes that should be addressed first:

1. If you expect interaction in your main effects here A and B, then testing for the significance of main effects doesn't make sense. (cf. an elegant paper by Dr. Venables first link )

2. I think doing sub-sampling doesn't help you (when I say help, I mean in the context of the number degrees of the freedom of F distribution which is used to test is treatment effect = 0). By sub-sampling, I mean replication at Block by Treatment level. In RCB, the degrees of freedom wasted in replication can only be used to test if the block effect is present or not. Using Blocks in expt. design is a "trick" to control for the extra variation which is not the main interest. In your case variance because of trays (block) is not of importance, so I don't know if it is useful to waste extra resources (ie money) in testing if the variance due to trays is significant or not.

3. I would be afraid to come to the conclusion that there was no block effect. The block effect in your case is random, and the test uses some "strong assumptions" to determine the sampling distribution (which may or may not be true depending on the strong assumptions). If your experiment started with RCB, please retain the structure of the data. Pooling of samples from block may confound your conclusions. You are again committing the error of accepting $H_o$ if you say there is no block effect (I am assuming you conclude no block effect by testing hypotheses. right?).

4. The point 4. further stresses my point 3. of choosing not to subsample, as essentially subsampling lets you test something (trays effect) which is not of primary interest to you. Even if this was of interest, you invest your money in replication to test something whose sampling distribution is not appropriately determined.

5. In the comment you say 12 treatments, but you only mention A and B as your treatment variables. Did you mean 12 levels for treatments A and B respectively? If yes, then you have only 2 treatments with 12 levels. If no, then you need to change the model.

For further details about subsampling in RCB, refer to chapter 3 of George Casella's design book.

Sincere apologies again, if I misunderstood you.