Multivariate post-hoc tests and multiple comparisons

Question

My research group is currently working on a medical imaging project in which we are studying brain perfusion in patients affected and unaffected by a particular type of injury using a technique called PWI-ASL (an MRI sequence). We have a machine learning technique for identifying regions in each subject's brain and calculating perfusion in those regions. We have two categorical independent variables (patient sex, disease state) with two levels each (male/female, affected/unaffected) and 62 brain regions (62 response variables). We want to compare perfusion in these regions between the four groups, with certain predetermined hypotheses about what will be higher, etc.

The problem we are running into is with post-hoc testing. We are consistently finding significant differences between the sexes and affected/unaffected individuals on MANOVA/adonis, etc., but by the time we perform any type of post-hoc testing, all differences are absolutely crushed by adjustments for multiple comparisons because we have so many with the 62 brain regions.

Someone in our group proposed using permutation testing (our data seem to fit its requirements) and performing permutation tests on each brain region individually, but should we not be adjusting the p values from each of the 62 comparisons?

If so, we are basically left with data that will never produce anything significant because of the multiple comparisons adjustments. Is there another way to analyze these data, or are we out of luck?

To summarize: 1. When performing permutation tests on multiple dependent variables, should the p values be adjusted for multiple comparisons? (I think so) 2. Does anyone have a recommendation for analyzing a large data set with multiple dependent variables that would not require such a massive multiple comparison adjustment?

Answer 1

Something else that you could try is to analyze your data using a linear mixed effects model which includes:

1. Fixed effects for patient sex and disease state;
2. Random effects for subject and brain region.

The outcome variable would consist of brain perfusion (presuming you can treat it as a continuous variable - from what I could see via Google, there may be multiple ways to quantify brain perfusion).

In R, you could fit this type of model with the following syntax:

m <- lmer(perfusion ~ sex * state + (1|region) + (1|patient), 
          data = yourdata, 
          REML= FALSE)

It is not clear from your answer whether disease state (aka state) is a predictor whose value (i) can change from one brain region to another or (ii) is the same across all brain regions.

If (i) is true, then you can add a random effect for state across regions by replacing (1|region) with (state|region) in the syntax provided above. If (ii) is true, then the above-stated model would be adequate.

You can then test contrasts (or comparisons) of interest based on the coefficients of the fixed effects in your stated model (i.e., the coefficients for the main effects of sex and state and their interaction).

See Robert Long's answer to this thread for the idea that inspired my response:

Mixed Effects Model with nested data.

I will tag Robert to my answer so that he can chime in as well.

Answer 2

1. Yes, whenever you test multiple hypotheses at once you should adjust for multiple comparisons.

2. Yes, there is another way. Try applying FDR control.

To elaborate, when there are a very large number of hypotheses, as seen in genomics, when we are controlling the probability of a single discovery we find that power is lost and nothing is significant.

Consider applying the method of False Discovery Rate (FDR) control, which is a much more liberal way of adjusting for multiple comparisons. I encourage you to search for resources. You need not have a strong statistics background to implement the BH algorithm. Background information can be found for instance here. The lectures from this course can also serve as reference if your stats background is a bit stronger.