# Correction for multiple testing

I am new to multiple testing. I do understand the general problematic and methods to correct for multiple testing (i.e. bonferroni), but I am not yet sure when to apply the rule.

I understand that it is most useful to display p-values of hypotests without correction and to mention that it is not yet done. Afterwards, when interpreting the p-value results, one should consider the correction. (e.g. https://www.graphpad.com/support/faq/when-to-not-correct-for-multiple-comparisons--startfragment---endfragment-/)

However, I am not sure about the general rules when you have to apply the correction. For example, in the same link above in the last example, they say correction is not nececary since the "The data from various demographic groups were then analyzed separately", "because the results are so consistent" and "ask the same basic question a different way, and all the comparisons point to the same conclusion". Can you explain why this needs no correction?

My interpretation: Rule of thumb from wikipedia: "Roughly speaking, the multiple comparisons problem arises whenever multiple hypotheses are tested on the same dataset (or datasets that are not independent) or whenever one and the same hypothesis is tested in several datasets." My interpretation is that "the data ... analyzed seperately" means literally different independent datasets, not only subgroups taken out of the complete previous dataset. But since they say they used "the same basic question", they still should have used correction according to wikipedia quote above.

Furthermore, is it okay to follow the general rule above from wikipedia, when to and when not to correct for multiple testing or do you have some literature/link/explanation what all the standard situations are?

As desired another example of applying a rule of thumb: i.e. Assume I want to collect data about cognitive biases in decision making. I have a data set consisting of randomly assign control (n=93) and treatment (n=97) participant. The control group receives for 6 different cognitive biases "control" texts, whereas the treatment group receives 6 "treatment" text. Each topic has separate questions, which are analysed. Each cognitive bias investigation produces a p-value by comparing control to treatment answers.

Interpretation with the rule of thumb from Wikipedia: I must correct for multiple testing, since I test multiple hypothesis on the same dataset (same participants used for different hypothesis). The fact that the hypotheses are testing different cognitive biases is no reason not to use the correction.

The Wikipedia rule of thumb in a nutshell:

Use correction when:

• Hypotheses on same dataset (or dependent data).

• Same hypothesis for different datasets.

• Personally, I find the main link (in your second paragraph) to unhelpful. Its terminology is very vague and generic--lacking precision. I'm unsure what you mean at the end by the 'general rule from Wikipedia'. // The Bonferroni method is approximate and can be much too conservative (since it is based on an inequality). When available, methods like Tukey's that are based on exact distributions and are intended for specific models, are preferable. // If you have a specific dataset and model in mind, it would be best to give details, so that one of us could recommend an appropriate approach. Mar 31, 2020 at 15:55
• Thank you BruceET for your comment. With 'general rule' I mean: I want to better understand when we have to correct for multiple testing. Ideally, a general rule of thumb would be helpful, i.e. the quote from wikipedia. // However, I take it from B. Bram's answer, that this problematic is very "depending", and a gerneal rule of thumb is hard to find. // Considering your last point I edit my question with an example. Apr 1, 2020 at 9:19
• Dear BruceET, I oppened another question for some more specific examples to not explode this thread. If you are interested you are welcome to pay a visit: stats.stackexchange.com/questions/461600/… Apr 22, 2020 at 9:53

The hard thing about these topics in statistics are that they seem very straight forward but when you have questions the answer is always it depends. When correcting for multiple tests this is the same. It all depends on the way you handle your data and interpret your results.

When looking at p-values for significance we in general accept the chance of 5% to have a type 1 error (rejecting null hypotheses falsely). So when looking at 20 tests that are all significant with a bit less than 0.05 alpha there is a big chance that one of them falsely rejects the null hypothesis. If these tests are done within or outside the same data set or even other research doesn't matter. If you're accepting the chance of 5% per test to have a type one error you're going to make a type one error eventually depending on the significance of the tests.

Let's have a simple example comparing doing tests to throwing dice. If throw a dice 2 consecutive times, what is the chance that I do not throw a 6 two times? 1 - 1/6 x 1/6 right? Now what If I throw a dice and then you throw a dice, do the odds change because two different persons throw the dice now ? No they don't. Same with statistical testing!

This is also a big problem in behavioral science when publishing research with barely significant results. Eventually you're going to publish research that had type 1 error.

This is why your statistical testing should always correct for multiple testing and have a solid theoretical base around them.

There are also other ways to correct for finding type 1 errors like several ways of cross validation.