Bonfferroni correction of correlation p-values. What's my k?

Question

I ran a series of Pearson correlations on my data and my supervisor told me to correct the p-values to account for the multiple comparisons I made. She told me to divide the p-value by the number of correlations I ran (k).

Now, I know it seems dumb (I'm pretty sure I'm just overthinking this) but I am not sure how many comparisons I have.

I am comparing scores of two different scales in two different groups of participants at two different points in time. However, I'm looking at correlations within each group and for both groups combined. Here's my correlation matrix: https://i.stack.imgur.com/4ZIDJ.jpg

Based on this, what would you guys say is my k (number of comparisons)?

Thanks!

Answer 1

I'm going to cause trouble, and give some pushback to the supervisor. It's always a good idea to argue against what your supervisor tells you. :)

You are probably aware of the reasoning behind adjusting p-values (or alpha values) in the case of multiple hypothesis tests. If you start with an alpha of 0.05, you are accepting a 5% chance of making a type-I error, that is of rejecting the null hypothesis when it is really true.† If you make multiple tests, by sheer mathematics, you inflate the chance of making this kind of error, unless you adjust your statistics to protect the false discovery rate (FDR) or familywise error rate (FWER).

The first piece of pushack I have is that Bonferroni correction is awfully conservative. I suspect in most cases people would rather not be that conservative with identifying significant results when using multiple tests, but they aren't aware of alternatives. You might look at some alternative correction procedures.

The second piece of pushback I have has to do with your goals of conducting multiple correlation analyses. With your goals in mind, is it better to be more liberal with identifying significantly correlated variables, or is there some purpose to protecting against identifying too many significant correlations? For example, if I were conducting multiple correlations as data exploration, or to check for colinearity, I wouldn't make any correction for multiple testing at all, because I would like err on the side of finding as many significant correlations as possible, even if I acknowledge that some may be "false positives". On the other hand, if I were doing mean separations among treatments, in most cases I would want to protect against spurious "false positives".

Also, consider the degree of correlation (r) vs. the information in conveyed by the p-value. Most of the correlations look pretty strong (r from about 0.5 to almost 1). Is the p-value in these cases very informative for your purpose? If the Bonferroni correction moves a correlation of r = 0.6 from significant to not significant, does this help you in your purpose or thwart it?

I honestly don't know the purpose of your correlation analysis. If it the final analysis, it may make sense to protect against inflated type-I error rates. But if it is an initial data exploration, it may not be beneficial. It depends how you are using the information the statistics are telling you.

This gets to one of the difficult things in the analysis of experiments. There isn't usually one right way. And different methods may lead to different answers. This can be frustrating and confusing. But it falls upon the analyst to understand the techniques as well as possible, and assess what is the best course depending on the goals of the analysis, and of course, always treating the data honestly.

---

† There's some simplification here. One, in the Neyman-Pearson view, the type-I error rate considers a long-term series of experiments. Two, what would "the null hypothesis is really true" mean when thinking about tests for correlation?

Answer 2

Make k equal to the number of hypothesis tests you ran. If you compare each value from the correlation matrix to zero inside the scope of project, count them all.

Answer 3

Let's say you have 4 scales, 2 groups and 1 time point. Then you have $\frac{4(4-1)}{2} = 6$ distinct correlations to compare. Assuming you are comparing the two groups to see if the correlations are equal, then you would want to divide by 6. If there were $k$ scales, you would divide by $\frac{k(k-1)}{2}$.

If you wanted to assess if the groups individually had statistically significant correlations, then you would assess each group's correlation matrix separately, and this would double the number of test, so divide by $2·\frac{k(k-1)}{2} = k(k-1)$. (Not sure why you would want to test if the aggregated sample correlation matrix on its own...but if you did, multiple by 3 instead of 2.)

If you have three times points, then you probably would want to assess the correlation for each scale at each pair of time points. In the case for 2 groups, 4 scales, and 3 time points, you would have $$3·\frac{4(4-1)}{2} + 4·\frac{3(3-1)}{2}$$ If you had $k$ scales and $n$ time points, you would have $$n·\frac{k(k-1)}{2} + k·\frac{n(n-1)}{2}$$ If you are interested in correlations between different scales at different time points, this would add more correlations (and more tests), but this wouldn't be common.