Can I investigate the simple main effects of a factor with multiple one-way ANOVAs if the main effects (from two-way ANOVA) were insignificant?

Question

I want to analyze the results of an experiment that was conducted as a 3 X 3 balanced, repeated measures study. There are two factors in my design--let's call them Factor A and Factor B. Both factors have three levels.

Factor A is the central factor being investigated in the study, and I am most interested in testing whether there are significant differences between the three levels in Factor A.

The main reason that I am including Factor B in my experimental design is because it is known (from previous studies) to have a significant effect on my response variable--for this reason, it wouldn't make sense to run a repeated measures study without including it as a factor. While we'd like to consider and report on the effects of Factor B, it isn't the central factor that we're investigating.

I decided to run a two-way repeated measures ANOVA on my data, and determined the following:

• There was no significant interaction effect between Factor A and Factor B.
• Factor B had a significant effect on my response variable (as expected, and indeed, this is why we bothered to include it as a factor in the first place).
• Factor A did not have a significant effect on my response variable.

Since there were no interaction effects between the two factors, I'm under the impression that I could proceed to run one-way ANOVAs on Factor B to better understand its simple main effects. However, this isn't the central focus of our study (and, again, has already been done in other studies).

The question that I have, is whether I can proceed to do any additional statistical analysis on Factor A, the factor that our investigation is most interested in? The two-way ANOVA has shown it to be insignificant--is that all I can report about it?

The idea that I had was to decompose the two-way repeated measures ANOVA into three separate one-way repeated measures ANOVA that would test the effect of Factor A within each level of Factor B. However, I don't know whether this is allowed or even valuable/meaningful.

If anyone has any insights into how to tackle this analysis, that would be helpful. Thanks!

Answer 1

Maybe you could get some information by 'collapsing' the design, but there are disadvantages:

• You are not analyzing the experiment you actually performed. In any report of findings you'd have to admit and justify collapsing the design.

• It could be that interaction and A effect are 'explaining' some of the variability---even if not significant. In that case error variance in the original full model might be smaller than error variance in the collapsed one, giving you a bit more power to detect any real differences among levels of factor B.

• What now appear to be 'replications' within levels of factor B are not necessarily anything like iid. If you do use a collapsed design, you'd have to take a careful look at its residuals.

My preference: Do ad hoc exploration of the original model via contrasts.