I conducted a mixed design ANOVA with two within-subjects factors: FactorA (2 levels), FactorB (2 levels), and one between-subjects factor: Group (2 levels). My main hypothesis regards the interaction FactorAFactorBGroup.
This is the ANOVA table:
Type III Repeated Measures MANOVA Tests: Pillai test statistic
Df test stat approx F num Df den Df Pr(>F)
(Intercept) 1 0.99424 4830.5 1 28 < 2.2e-16 ***
Group 1 0.01375 0.4 1 28 0.53715
FactorA 1 0.46649 24.5 1 28 3.197e-05 ***
Groups:FactorA 1 0.00685 0.2 1 28 0.66367
FactorB 1 0.14451 4.7 1 28 0.03825 *
Group:FactorB 1 0.15108 5.0 1 28 0.03378 *
FactorA:FactorB 1 0.09930 3.1 1 28 0.08985 .
Group:FactorA:FactorB 1 0.02737 0.8 1 28 0.38232
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
As you can see, there is a significant GroupFactorB effect, but the interaction of my interest (GroupFactorA*FactorB) is not significant.
However, if I try to conduct a post-hoc analysis on the GroupFactorAFactorB:
my_data$Interaction <- interaction(my_data$FactorA, my_data$FactorB)
my_data$Interaction <- factor(my_data$Interaction)
post_hoc_model <- lme(value~interaction, random=~1|Subject/FactorB/FactorB,data=my_data)
planned_post_hoc <- summary(glht(post_hoc_model,linfct=mcp(interaction=c(
"Control.Level1.Level1-Control.Level2.Level1==0",
"Experimental.Level1.Level1-Experimental.Level2.Level1==0",
"Control.Level1.Level2-Control.Level2.Level2==0",
"Experimental.Level1.Level2-Experimental.Level2.Level2==0"))),
test=adjusted("BY"))
I obtain these results:
Linear Hypotheses:
Estimate Std. Error z value Pr(>|z|)
Control.Level1.Level1-Control.Level2.Level1==0 0.053125 0.019228 2.763 0.0477 *
Experimental.Level1.Level1-Experimental.Level2.Level1==0 -0.018750 0.027192 -0.690 1.0000
Control.Level1.Level2-Control.Level2.Level2==0 0.003125 0.019228 0.163 1.0000
Experimental.Level1.Level2-Experimental.Level2.Level2==0 -0.025000 0.027192 -0.919 1.0000
Specifically, the first comparison is significant, after correcting for False Discovery Rate, and this is coherent with my hypothesis and with the simple inspection of the means.
Probably, the interaction was not significant due to my small sample sizes.
I know that, after a not-significant interaction, one does not have to do post-hoc for that. So I'm wondering how much this methodology is criticable by a referee. I thought to specify that the post-hoc comparisons should be interpretate with caution because the interaction was not significant. Moreover, I found this practice in a lot of researches.
Do you think this way of proceeding is totally wrong?
How can I eventually defend my analysis from a referee critique (maybe by referring to published statistical papers)?
Interaction
rather thaninteraction
? I fixed it now. $\endgroup$