# What is the definition of dataset (for Bonferroni purposes)?

I'm having difficulties finding a clear rule for when a series of test should be considered a multiple comparison and when we should apply p-value corrections (like Bonferroni).

I understand corrections must be applied every time multiple hypothesis are tested using the same dataset. A classical example is a Post-hoc Tukey test on data from an ANOVA.

However, what is the proper definition of a "dataset"? Whenever two tests share the a sample, are they the same dataset? Do they need to share all samples? The tests must share the same hypothesis?

I found many questions related to mine in this forum and online, but all of them seem to handle examples. If their particular case is or is not a multiple comparison and whether it needs correction, but none seem to come with a objective definition of "dataset".

• For perspectives questioning the utility of such adjustments, see Rothman, K. J. (1990). No Adjustments Are Needed for Multiple Comparisons. Epidemiology, 1(1), 43–46. doi.org/10.1097/00001648-199001000-00010 and Saville, D. J. (1990). Multiple Comparison Procedures: The Practical Solution. The American Statistician, 44(2), 174–180. Retrieved from jstor.org/stable/2684163 – Heteroskedastic Jim Dec 14 '18 at 17:47

It's assumed (incorrectly) that data in different datasets must, by design, be independent whereas two tests calculated with the same dataset must be dependent (also not necessarily correct). To justify and discuss the type of testing correction to use, one should consider the "family of tests". If the tests are dependent or correlated (that is to say that the $$p$$-value of one test actually depends on the $$p$$-value from another test), Bonferroni will be conservative. (NB: some rather dicey statistical practices can make Bonferroni anti-conservative, but that really boils down to non-transparency. For instance: test main hypothesis A. If main hypothesis non-significant, test hypotheses A and B and control with Bonferroni. here you allowed yourself to test B only because A was negative, this makes tests A and B negatively correlated even if the data contributing to these tests are independent.)