# When do I need to care about multiple comparisons

I have a study design and have been asked to test four hypotheses. I have a fairly good understanding of multiple comparisons when it comes to the same hypothesis i.e. if I have 3 levels then I need to correct for 3 comparisons, but am unsure whether I need to perform an extra step as I have multiple hypotheses tied around the same dataset.

To summarise what each hypothesis is asking.

H1 = is assessing how often a certain response is made following an event (4 groups)

H2 = is assessing whether the result following the event is the same of different between the four aformentioned groups.

H3 = assesses across all of those groups whether the time taken before event occurred is statistically different from another set of data.

H4 = assesses whether result following the event is statistically different from zero. One of the groups is omitted during this analysis because the data is appropriate for H2 but not for H4.

So all my hypotheses draw from the same population, but H1 draws from it in a completely different manner from any other, as does H3. H4 is using a subset of the data from H2 but is asking a different global question about the event whilst not caring about groups.

To make things more interesting, for H2, H3 and H4, each analysis was carried out assessing the result following the event after ten business days and also after 126 business days.

All my statistics have been corrected in a reasonable fashion within each hypothesis. However when looking at them together I am wondering whether it is sufficient that they are asking different questions or is the fact that they use the same underlying data an issue that requires correction?

Imagine you do not test 4 but 10000 hypotheses. Just by chance, a few of those will be false positive. It does not matter if they test the same or different hypotheses, so yes, you do need to correct for multiple comparisons.

• Okay, at the moment my statistics are corrected within each individual test and post hocs using a Bonferroni correction (p / number of comparisons). Is this as simple as changing the divisor to the total number of tests run throughout the analysis? Oct 18 '19 at 13:49

You would need to correct for having four hypotheses in order to maintain the family wise error rate: https://en.wikipedia.org/wiki/Family-wise_error_rate

Since there is an overwhelming drive to interpret statistically significant findings as being well true, it is important to ask the following question:

What is the chance that one of my significant hypotheses is a false positive?

When you only have one hypothesis then your false positive rate is just your significance level. However, when you have more than one then the chance of having one of the significant results being a false positive goes up. So yes you need to correct for it.

Since you are already correcting within your hypotheses for multiple comparisons I believe the correct approach is to just multiply the divisor in your Bonferroni correction by the number of tests you are doing.

• So (excluding the actual ANOVAs), the number of pairwise tests is 22 (h1 = 6, h2 = 6*2, h3 = 1*2, h4 = 1*2, == 7 hypotheses). Should I be correcting everyone of these tests by using 22 or multiplying each which is already corrected within its hypothesis by 7? Oct 18 '19 at 14:41
• Since each of your defined hypotheses actually has many subcomponents you should correct by the total number of tests you are doing. So everyone corrected by 22. This is because from a false positive perspective you are actually evaluating 22 hypotheses (you just grouped them into 4 categories by calling them h1 through h4). Oct 18 '19 at 14:44
• Thanks! (I'm so lucky that my p values are low) Oct 18 '19 at 14:47
• That's awesome! Oct 18 '19 at 14:48