I am not 100% sure this belongs here, but it is a stats question. I am reviewing a paper that conducts a 2-way PERMANOVA with interaction and performs a post hoc test. My problem is arising from the reporting of the results of the posthoc test. They tested the general model (distance matrix)~(TIME)*(TREATMENT). They found that TIME and TREATMENT were significant, but the interaction of TIME:TREATMENT was not significant. Their post hoc testing should then be restricted to differences within time and differences within treatment.
But their results table presents grouping differences (a,b,c, etc) for treatment at each level of time and differences for time at each level of treatment. Which, to me, regardless of whether or not is is a PERMANOVA or a mere ANOVA is a test of the interaction. I put this point to the authors in the first round of revisions and they responded with 'no, we did it right.'
So, my questions are
I don't think I am missing a basic premise of post hoc testing, but did I miss some change in the basic premise of post hoc testing in the past few years?
They are using Primer-e, which I have no experience with. When you push the PERMANOVA post hoc test button on this software, does it run all combinations/contrasts of all independent variables regardless of their significance or does it restrict itself to testing only the significant ones?
Assuming the problem isn't me, I am trying to see the source of the problem so I can help the authors fix it because I like the manuscript.