why correct for multiple comparisons in a correlation table?

Question

I often read that in a correlation matrix the p-values for all correlations in a NxN matrix should be adjusted for multiple comparisons (for example: Multiple comparisons for correlation matrix?). Most of my co-authors also adamantly believe this but nobody is able to give me a source for this. My question is: do I have a fundamental misunderstanding of why we need to correct for multiple corrections to reduce type-1 error?

My understanding is that if we want to test a hypothesis with an alpha of 0.05, that within this hypothesis the nested set should have a confidence of 95%. So for example:

H0 = $\mu_{1} + \mu_{2} + \mu_{3} = 0$
H1 = $\mu_{1} + \mu_{2} + \mu_{3} \neq 0$

The full set should have a reliability of 95%. This is why when examining pairs causes type 1 error inflation, because the set of pairs:

$\mu_{1} = \mu_{2}$
$\mu_{1} = \mu_{3}$
$\mu_{2} = \mu_{3}$

All are subsets of the original nulhypothesis and the entire set should have reliability of 95%. To maintain this, a correction on the alpha (or p-values) for every pairwise comparison is needed to maintain 95% confidence.

Is this understanding correct? Because its implication would be that multiple comparison should occur within a nested set of comparison, but should not occur between non-nested comparisons? So if I have a new nulhypothesis about a different Y, than that nulhypothesis does not be corrected for the existance of the nulhypothesis test on the other dependent variable.

So, how are all bivariate correlations in a NxN matrix subject to multiple comparisons? Is this because the same variables are repeatly correlated with other variables and so form a nested set of their own? I’m really struggling with this, so if someone has a good (mathematical) paper for me to help me understand the fundamentals that would be awesome.

Answer 1

Your understanding is not exactly incorrect; this is an area where as Jacob Cohen put it in his book Multiple correlation/regression analysis "reasonable people can differ".

And it's not answerable from a strictly statistical point of view. It's a question of which set of tests you want to have at 0.05 chance. One hypothesis? One correlation table? One paper? One set of related hypotheses? You can control error rate at any of those levels.

However, if your correlation table were actually all ones where the null of no correlation in the population was true, then, if you don't correct for the number of correlations, about 5% of the ones in the table will be significant. So, maybe you should correct by table? On the other hand 1) Often, we know that the null is false in the population and 2) By correcting for multiple comparisons, we lower power.