I am new to multiple testing. I do understand the general problematic and methods to correct for multiple testing (i.e. bonferroni), but I am not yet sure when to apply the rule.
I understand that it is most useful to display p-values of hypotests without correction and to mention that it is not yet done. Afterwards, when interpreting the p-value results, one should consider the correction. (e.g. https://www.graphpad.com/support/faq/when-to-not-correct-for-multiple-comparisons--startfragment---endfragment-/)
However, I am not sure about the general rules when you have to apply the correction. For example, in the same link above in the last example, they say correction is not nececary since the "The data from various demographic groups were then analyzed separately", "because the results are so consistent" and "ask the same basic question a different way, and all the comparisons point to the same conclusion". Can you explain why this needs no correction?
My interpretation: Rule of thumb from wikipedia: "Roughly speaking, the multiple comparisons problem arises whenever multiple hypotheses are tested on the same dataset (or datasets that are not independent) or whenever one and the same hypothesis is tested in several datasets." My interpretation is that "the data ... analyzed seperately" means literally different independent datasets, not only subgroups taken out of the complete previous dataset. But since they say they used "the same basic question", they still should have used correction according to wikipedia quote above.
Furthermore, is it okay to follow the general rule above from wikipedia, when to and when not to correct for multiple testing or do you have some literature/link/explanation what all the standard situations are?
As desired another example of applying a rule of thumb: i.e. Assume I want to collect data about cognitive biases in decision making. I have a data set consisting of randomly assign control (n=93) and treatment (n=97) participant. The control group receives for 6 different cognitive biases "control" texts, whereas the treatment group receives 6 "treatment" text. Each topic has separate questions, which are analysed. Each cognitive bias investigation produces a p-value by comparing control to treatment answers.
Interpretation with the rule of thumb from Wikipedia: I must correct for multiple testing, since I test multiple hypothesis on the same dataset (same participants used for different hypothesis). The fact that the hypotheses are testing different cognitive biases is no reason not to use the correction.
The Wikipedia rule of thumb in a nutshell:
Use correction when:
Hypotheses on same dataset (or dependent data).
Same hypothesis for different datasets.