1
$\begingroup$

So I had a question regarding multiple comparisons. I was looking at 4 brain networks, with regions of interest (ROIs) that were defined (ranging from 3 to 7 ROIs) to study depression. I extracted these ROIs from ICA (independent component analysis). I then extracted the IC that most matched the network of interest. Then I set a threshold to get individual ROIs within the network. I then did a pairwise statistical analysis using a t-test for each pair of ROIs within each network (I did not perform between network analysis). So I ended up with a matrix of "functional connectivity values" for each network (i.e. 7x7 matrix). Now, I was wondering if it was necessary to perform a multiple comparisons test on my data? I know some literature performs a whole brain analysis randomly and I see why they would need to adjust for the type I error, but seeing that I am not doing that, would that still apply or could it be arguable? Is there some way I can justify not performing a multiple comparisons correction? Also I was wondering what your thoughts about this article are? https://www.graphpad.com/support/faq/when-to-not-correct-for-multiple-comparisons--startfragment---endfragment-/

Another question..If I am performing a t-test for each pair of ROI's (6 ROIs in a network) would I adjust the p-value to be 0.05/15? Or would I break up the bonferroni correction to only be adjusted for all the ROI pairs matching with ROI1? What I mean by that is, in a matrix I'll have 1 2 3 4 5 6 1x 2 x 3 x 4 x 5 x 6 x That way I'll only correct for the ROI1 duplicates, instead of all 6 ROI pairwise combinations. I hope that makes sense.

$\endgroup$
1
3
$\begingroup$

Any time you do multiple tests at the 5% level, you have probability $0.05$ of making a 'false discovery'. That is. you have a chance of rejecting a null hypothesis (no difference) that is true. So if you do all ${7 \choose 2} = 21$ comparisons of pairs of ROI's, you would expect--by chance alone--to make about one false discovery among the 21 comparisons. One says that the error probability is 5% for each individual comparison, but the error probability is higher for the 'family' of comparisons.

There are very many different methods of controlling the family error rate in such situations. They depend on the number and kind of comparisons made. A few of these methods are Fisher's LSD (least significant difference) method, Tukey's HSD (for honest significant difference), Student-Newman-Kuels (SNK) method, and so on. The Bonferroni method is based on an inequality (in which you are assuming the worst case) and so it can be a little too 'conservative' (too reluctant to declare significant differences).

If you will do only a few ad hoc comparisons, it is worthwhile finding a method in which the criterion for rejection matches the number of comparisons. For example, you may have three groups in a one-way ANOVA. The null hypothesis is that all three have the same mean. If the overall F-test rejects the null hypothesis, you know that there are some differences among the three, but not how many or which ones.

If all groups have the same sample size, it seems pretty sure that the groups with the largest and smallest means must be different. But then you need to test whether the middle one of the three differs from the smallest, from the largest, or from both. You can explore this with two ad hoc tests, and using the Bonferroni method, the 5% family error rate would not be exceeded if you did both of the ad hoc tests at the 2.5% level.

Another scenario arises when there are several (say 3) treatment groups and one control group. If you are interested to know only which treatment groups do better than the control group, you can explore this with only three tests (oh well--perhaps a fourth to see whether one of two especially promising treatments is significantly better than the other).

In general, if you have $k$ groups, you will not necessarily be able to say for sure exactly which groups really differ from which others. For example with $k = 6,$ maybe the best you can do is to say that the two groups with the largest means are significantly different from the other four. But tests may not be able to determine whether the top two really differ from each other or whether there are real differences among the bottom four.

An initial study, before a multi-group experiment is started, ought to determine how many replications are needed in each group in order to have a reasonable probability of detecting differences of a particular size among the various groups.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.