0
$\begingroup$

In R, posthoc.quade.test (package PMCMR) or quadeAllPairsTest (the new variant, PMCMRplus) are posthoc tests for Quade test and offer the option for adjustments (i.a. Holm, BH etc.).

In my previous understanding, posthoc tests are applied after obtaining a significant omnibus test (unless already implemented in the posthoc test, but this does not seem the case from the PMCMRplus desription PDF, https://cran.r-project.org/web/packages/PMCMRplus/index.html), and are basically designed to avoid pairwise testing + correcting afterwards (may be wrong, see question):

"If no p-value adjustment is performed (p.adjust.method = "none"), than a simple protected test is recommended, i.e. all-pairs comparisons should only be applied after a significant quade.test.".

My question: what is the advantage/difference of (1) test-specific (Quade) post-hoc test over (2) simple pairwise testing with any other test (e.g. Wilcoxon), if adjustments (Holm, BH) are required anyway? Wouldn't this only add unnecessary complexity to our statistical methods? Are these "basic conceptual" differences between (1) and (2) (i.e. test different questions), or rather "performance" differences (i.e. are more/less sensitive, error-prone, calculation time...)?

Maybe I completely miss the point, but happy to learn!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

My understanding is that some post-hoc tests require that there be a significant omnibus test prior, and others don't. (See here for some discussion).

In general, it is better --- that is, statistically sounder --- to avoid pairwise tests, and instead to embrace post-hoc tests that are fashioned for the purpose. The problem with pairwise tests, as I see it, is that each comparison between two groups ignores the rest of the data. In the case of pairwise Wilcoxon-Mann-Whitney tests, there can be a particular problem, exemplified by the Schwenk dice phenomenon, where, from a practical point of view, the results are incommensurate with each other.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Magnifico Yes, thanks, I completely agree - so why should Bonferroni/Holm/whateveer correction be necessary after omnibus + post-hoc test, as suggested in the cuted help file? (at least I understand "...If no p-value adjustment is performed (p.adjust.method = "none")..." in the way that it would be necessary? This is what I was confused and asked the question. $\endgroup$
    – Martin
    Sep 1, 2018 at 19:33
  • $\begingroup$ Some post-hoc tests control for the familywise error rate by their design. These include Tukey HSD and Dunnett. Others basically make multiple comparisons, for which a p-value adjustment might apply. I suspect --- and I'm speculating here --- that the quoted text is using the omnibus test as a "protection" of type 1 error, that the omnibus test indicates there was at least one significant pairwise difference. This is the same way that "protected" is used in Fisher's protected LSD. $\endgroup$
    – Sal Mangiafico
    Sep 3, 2018 at 23:45
  • $\begingroup$ @ Sal Mangiafico: May I ask you to add the fact (as far as I understand your comment) that some, but not all post-hoc-tests control for FWER by design, and a list or link, WHICH of these post-hoc-test do and do not? I was not able to find this important information (and implicitly - but obviously in error - hat believed that they all do) - happy to have this as an answer! $\endgroup$
    – Martin
    Sep 5, 2018 at 18:26
  • $\begingroup$ @Martin, I don't have a such a list. The ones that come to mind that control FWER are Tukey and Dunnett. I suspect with some others, there might be some theoretical discussion, like what the "protection" in the protected LSD counts for. Such a discussion are far beyond me. $\endgroup$
    – Sal Mangiafico
    Sep 6, 2018 at 0:17
  • $\begingroup$ To give an example of the other kind of test, as far as I understand, Dunn test 1964 as post-hoc for Kruskal-Wallis is essentially the same as conducing pairwise Mann-Whitney tests, except that the original rankings of the dependent variable from the K-W test are preserved. So in this case, there are simply multiple related hypotheses, which call for a p-value adjustment. (No guarantees that this understanding is completely correct.) $\endgroup$
    – Sal Mangiafico
    Sep 6, 2018 at 0:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.