Significant interaction but non-significant pairwise comparisons

Question

I ran a 3-way ANCOVA (3 within x 3 within x 2 between) and ended up with a significant 3-way interaction (p=0.047, so the significance wasn't large to begin with). I ran additional 3 2-way ANCOVAs for my first 3 within factor, and ended up with a significant 2-way interaction in one of the levels (p=0.012). I described this result as 'significant' after Bonferroni-adjusting the p-values (i.e. comparing with 0.05/3=0.0167), which justified me to proceed with pairwise comparisons (i.e. 3 1-way ANCOVAs for each second within-subjects level).

The pairwise comparisons for that particular level returned me with p-values, .054, .07, and .077. These seem 'marginally significant' but when accounting for multiple comparisons adjustment, they are not (0.0167/3=0.0056). However, I feel that since I got significant interactions earlier I need to explain explicitly where in particular the interactions arose from.

My question is, are the p-values .054, .07, and .077 enough for me to explain the 2-way interaction, i.e. the differences may be where the interaction occurred but the differences were too small to yield any 'significant' effects? (apologies for saying 'significant' too much - I try avoiding using the whole p-value cutoff=0.05 thing but my academic field still practices it and I'm merely a student working on my thesis).

I feel that getting non-significant p-values still should not stop me from explaining my interactions, which are informative in terms of the aim of the experiment. What are your thoughts? Also, if you know any literature that specifically talks about issues that are similar to what I described, I would be very interested to have a read. Thank you!

Answer 1

Is the actual size of the interaction effect large enough to be of practical importance?

If not, I'd mention in a footnote that there is a marginally significant but unimportant Interaction effect (P-value 4.7%).

If so, I'd say the interaction is just barely significant at the 5% level and, while ad hoc tests are not significant at 5%, they suggest interaction may be due to [whatever] and mention possible effect size.

Often multifactor ANCOVAs have sufficient sample size (number of replications) to give decent power for detecting main effects, but not for resolving interaction effects.

Also, I'd do post hoc diagnostics of residuals for normality and equal variances with extra care. Sometimes 'marginally significant' interactions turn out to be artifacts of unmet assumptions.