Why does power reduce when increasing the number of post-hoc comparisons?

I'm performing a Dunn's post test comparison of my data after a Kruskal-Wallis test (4 groups, treated and control each one has drug and vehicle injection).

The comparisons I've been doing are

Treated Vehicle vs Control Vehicle (I want to see the vehicle doesn't affect the treatment)
Treated Drug    vs Control Drug (I want to see the drug affects the treatment)
Control Vehicle vs Control Drug (I want to see that both control are the same)

I was wondering if the other comparisons were also possibly significant but when I increased the number of comparisons I lost significance in the ones I had before and everything was "ns" (not significant).

I looked for the "test help" (I use GraphPad Prism) and read that if you increase the number of comparisons it will be harder to resolve differences because it divides the alpha you choose between the n number of comparisons but I don't understand why is this happening.

When I say "why is this happening", I mean I want to understand intuitively why it is necessary to divide the given alpha.

My intuition tells me that if two things were significant before they must continue to be significant. Otherwise if we suppose that the "better" comparison is everybody against each other, then we should only trust that comparison because the other comparisons have some kind of "masking effect".

The problem is probably best known as the Multiple Comparisons Problem.

When you choose a significance level for a given comparison you are allowing for a certain Type I error rate. If I perform multiple comparisons (among multiple groups as you wrote above) I am necessarily increasing the likelihood that I make a Type I error. At the extreme end if I have two identical groups but $100$ metrics I'm testing in one sample from each group then $\alpha$ fraction of those will come back erroneously significant. But at $100$ metrics I'm almost guaranteed to make an error every time.

One possible correction for this is to decrease $\alpha$ such that as I increase the number of comparisons I maintain the desired false positive rate. From the previous example, if instead of an $\alpha$ significance level I modify it to an $\alpha / 100$ level with my $100$ metrics comparisons I can recover my desired false positive rate. Intuitively if $\alpha$ gives me a $5\%$ false positive rate then $\alpha / 100$ should give me a $0.05\%$ false positive rate so when I perform my $100$ tests I will recover the overall significance level I desired.

This correction, that GraphPad is doing and outlined above, is known as the Bonferroni Correction. It is derived by adjusting the significance level of a multiple testing scenario with the requirement that the familywise error rate (the notion of type I error) is kept at the ceiling of the single comparison error rate.

The intuition in changing $\alpha$ is bound up in the notion that the per-test and familywise error rates are different in multiple testing scenarios and what we desire is a constraint on the familywise.

• I understand that you want to keep the alpha of the whole family and that you must do this by dividing. However, aren't you asking for the same values that be "differently different" because of the comparisons you're trying to make? The calculation is the same, you only move the threshold accordingly to your number of comparissons. – Matias Andina May 27 '15 at 1:18
• Just so I'm clear, are you trying to reconcile why doing a single metric test needs significance level of $\alpha$ but that very same test in a larger family of tests needs $\alpha / n$ when nothing about the data in that test has changed? – cwharland May 28 '15 at 3:25
• Not exactly, I'm trying to understand why is conceptually correct. For example, A=1 B=2 C=5, ANOVA<0.05. You do comparison suppose A vs B is p<0.05 when you do only that so you must think they are different. Then you add A vs C and B vs C and A vs B is no longer "different" just because you looked "too many times" in other places. I understand the math, the concept is messing up with my head because it seems to me that one should then mandatory use every comparison possible always (otherwise you end up choosing what comparisons are you interested in) – Matias Andina May 28 '15 at 21:38