0
$\begingroup$

I am new to multiple testing. I do understand the general problematic and methods to correct for multiple testing (i.e. bonferroni), but I am not yet sure when to apply the rule.

I understand that it is most useful to display p-values of hypotests without correction and to mention that it is not yet done. Afterwards, when interpreting the p-value results, one should consider the correction. (e.g. https://www.graphpad.com/support/faq/when-to-not-correct-for-multiple-comparisons--startfragment---endfragment-/)

However, I am not sure about the general rules when you have to apply the correction. For example, in the same link above in the last example, they say correction is not nececary since the "The data from various demographic groups were then analyzed separately", "because the results are so consistent" and "ask the same basic question a different way, and all the comparisons point to the same conclusion". Can you explain why this needs no correction?

My interpretation: Rule of thumb from wikipedia: "Roughly speaking, the multiple comparisons problem arises whenever multiple hypotheses are tested on the same dataset (or datasets that are not independent) or whenever one and the same hypothesis is tested in several datasets." My interpretation is that "the data ... analyzed seperately" means literally different independent datasets, not only subgroups taken out of the complete previous dataset. But since they say they used "the same basic question", they still should have used correction according to wikipedia quote above.

Furthermore, is it okay to follow the general rule above from wikipedia, when to and when not to correct for multiple testing or do you have some literature/link/explanation what all the standard situations are?

As desired another example of applying a rule of thumb: i.e. Assume I want to collect data about cognitive biases in decision making. I have a data set consisting of randomly assign control (n=93) and treatment (n=97) participant. The control group receives for 6 different cognitive biases "control" texts, whereas the treatment group receives 6 "treatment" text. Each topic has separate questions, which are analysed. Each cognitive bias investigation produces a p-value by comparing control to treatment answers.

Interpretation with the rule of thumb from Wikipedia: I must correct for multiple testing, since I test multiple hypothesis on the same dataset (same participants used for different hypothesis). The fact that the hypotheses are testing different cognitive biases is no reason not to use the correction.

The Wikipedia rule of thumb in a nutshell:

Use correction when:

  • Hypotheses on same dataset (or dependent data).

  • Same hypothesis for different datasets.

Improve this question
$\endgroup$
3
  • $\begingroup$ Personally, I find the main link (in your second paragraph) to unhelpful. Its terminology is very vague and generic--lacking precision. I'm unsure what you mean at the end by the 'general rule from Wikipedia'. // The Bonferroni method is approximate and can be much too conservative (since it is based on an inequality). When available, methods like Tukey's that are based on exact distributions and are intended for specific models, are preferable. // If you have a specific dataset and model in mind, it would be best to give details, so that one of us could recommend an appropriate approach. $\endgroup$
    – BruceET
    Commented Mar 31, 2020 at 15:55
  • $\begingroup$ Thank you BruceET for your comment. With 'general rule' I mean: I want to better understand when we have to correct for multiple testing. Ideally, a general rule of thumb would be helpful, i.e. the quote from wikipedia. // However, I take it from B. Bram's answer, that this problematic is very "depending", and a gerneal rule of thumb is hard to find. // Considering your last point I edit my question with an example. $\endgroup$
    – Onerock
    Commented Apr 1, 2020 at 9:19
  • $\begingroup$ Dear BruceET, I oppened another question for some more specific examples to not explode this thread. If you are interested you are welcome to pay a visit: stats.stackexchange.com/questions/461600/… $\endgroup$
    – Onerock
    Commented Apr 22, 2020 at 9:53

1 Answer 1

Reset to default
2
$\begingroup$

The hard thing about these topics in statistics are that they seem very straight forward but when you have questions the answer is always it depends. When correcting for multiple tests this is the same. It all depends on the way you handle your data and interpret your results.

When looking at p-values for significance we in general accept the chance of 5% to have a type 1 error (rejecting null hypotheses falsely). So when looking at 20 tests that are all significant with a bit less than 0.05 alpha there is a big chance that one of them falsely rejects the null hypothesis. If these tests are done within or outside the same data set or even other research doesn't matter. If you're accepting the chance of 5% per test to have a type one error you're going to make a type one error eventually depending on the significance of the tests.

Let's have a simple example comparing doing tests to throwing dice. If throw a dice 2 consecutive times, what is the chance that I do not throw a 6 two times? 1 - 1/6 x 1/6 right? Now what If I throw a dice and then you throw a dice, do the odds change because two different persons throw the dice now ? No they don't. Same with statistical testing!

This is also a big problem in behavioral science when publishing research with barely significant results. Eventually you're going to publish research that had type 1 error.

This is why your statistical testing should always correct for multiple testing and have a solid theoretical base around them.

There are also other ways to correct for finding type 1 errors like several ways of cross validation.

I hope this answers your question!

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks to B. Bram for your contribution! I do understand your argumentation. But on that basis, wouldn't we have to correct every possible hypotest on the same topic that has ever been done when we do a new test? $\endgroup$
    – Onerock
    Commented Apr 1, 2020 at 10:15
  • $\begingroup$ Wait, I'm a little slow. Based on your last statement: When I do research on a topic, should I look for all empirical studies with hypothese tests on that topic when doing literature research and correct for them in my conclusion? I have never seen anyone do that. // Furthermore, in this case I would also have to correct my work after every new additional topic on the subject that comes after my paper. I have also never seen anybody doing that. $\endgroup$
    – Onerock
    Commented Apr 2, 2020 at 10:00
  • $\begingroup$ I'm sorry I see now my answer is confusing. Indeed normally someone wouldn't correct for multiple testing when done in previous research. I'll edit my comment I was trying to explain something else but i see i didn't do a good job there. $\endgroup$
    – B. Bram
    Commented Apr 2, 2020 at 12:42
  • 1
    $\begingroup$ To answer your first comment, No you don't need to correct for every research you are using. When ten studies indicate the same relation your chance of type 1 error actually decreases. (because implicitly your sample size increases). However you should be aware that when citing barely significant research on different topics eventually you're going to cite an article that had type 1 error. Therefore we always empirically test our hypotheses when possible. $\endgroup$
    – B. Bram
    Commented Apr 2, 2020 at 12:45
  • $\begingroup$ I see, that is useful, thank you! $\endgroup$
    – Onerock
    Commented Apr 3, 2020 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.