I have seen some guides on how to craft your language when reporting statistical results (e.g., Smith (2020) https://doi.org/10.1007/s12237-019-00679-y) but does anyone know how to report multiple comparisons? Especially if you have numerous comparisons to talk about (ie > 10), it would be impractical to use boxplots.

  • 1
    $\begingroup$ Rather than closing your question, I have edited out the part that is unsuitable for our site. It would help if you could clarify what you mean by "report," because one can imagine many possible answers, ranging from extensive explanation through simple Bonferroni adjustments of the p-values. $\endgroup$ – whuber May 14 '20 at 16:25
  • $\begingroup$ Why was it unsuitable? $\endgroup$ – vanguard605 May 14 '20 at 20:39
  • $\begingroup$ It invited dispute and opinion without framing a specific, answerable question. $\endgroup$ – whuber May 14 '20 at 20:41
  • $\begingroup$ Remember: Stack Exchange is for Q&A, not a discussion forum. And if you didn’t know that, now you do :) $\endgroup$ – Dave May 14 '20 at 20:53

All current approaches to handling multiple comparisons focus on (read: are equivalent to) using significance tests and modifying thresholds based on ranked unadjusted p-value, the overall number of tests, and either the prespecified family-wise error rate or the number of false discoveries, etc. It seems necessary to invoke null hypothesis significance testing to some degree to handle multiplicity.

The chief complaint against a p-value is that it is not quantitatively useful in its own right. To summarize findings, (and 10 comparisons is not a lot), you can use a simple forest plot with 95% CIs. These are powerful visual tools and convey a large quantity of information in a readily digestible format, Standardize the estimated effect then show significance in terms of the magnitude and precision of effect according to the ranked test statistics for each comparison.

  • $\begingroup$ Thanks for the tip I have up to 21 comparisons but I think your forest plot idea maybe useful. Had not heard of them before so thanks. $\endgroup$ – vanguard605 May 14 '20 at 20:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.