I'm performing a Dunn's post test comparison of my data after a Kruskal-Wallis test (4 groups, treated and control each one has drug and vehicle injection).
The comparisons I've been doing are
Treated Vehicle vs Control Vehicle (I want to see the vehicle doesn't affect the treatment)
Treated Drug vs Control Drug (I want to see the drug affects the treatment)
Control Vehicle vs Control Drug (I want to see that both control are the same)
I was wondering if the other comparisons were also possibly significant but when I increased the number of comparisons I lost significance in the ones I had before and everything was "ns" (not significant).
I looked for the "test help" (I use GraphPad Prism) and read that if you increase the number of comparisons it will be harder to resolve differences because it divides the alpha you choose between the n number of comparisons but I don't understand why is this happening.
When I say "why is this happening", I mean I want to understand intuitively why it is necessary to divide the given alpha.
My intuition tells me that if two things were significant before they must continue to be significant. Otherwise if we suppose that the "better" comparison is everybody against each other, then we should only trust that comparison because the other comparisons have some kind of "masking effect".