# Multiple comparisons - adjusting p values when some variables are correlated

I'm working with 400+ variables (mostly 2D gel data) across 2 conditions for which I've calculated p values with a students t-test. Some of the variables are highly correlated. I'm wanting to adjust the p values to allow for multiple comparisons and I've used p.adjust in R with the "BH" method but I understand that it supposed to be for independent variables. I also tried the "BY" method as I thought being "under dependency" meant that it allowed for some variables to be non-independent (ie correlated), and so I assumed the adjusted p values would be lower than with the "BH" method, but they're not. (When I use the p.adjust BY method on my data the adjusted p values are all 1). Is there a method of adjusting p values that allows for some correlated variables? Thanks for any help.

> pvals$adjusted_BY <- p.adjust(pvals$p_value, method = "BY")
> pvals$adjusted_BH <- p.adjust(pvals$p_value, method = "BH")
1 2366 0.001361787           1   0.5828448
2  746 0.008377390           1   0.9516818
3 1635 0.017267163           1   0.9516818
4 1577 0.025868990           1   0.9516818
5 1588 0.029862316           1   0.9516818
6  891 0.036119770           1   0.9516818

• Could you provide a minimal code example? That might help get a better answer. – Fabian Fagerholm May 27 '16 at 6:18
• "adjust the p values" How did you derive the p-values? What kind of test did you conduct? – Roland May 27 '16 at 7:08
• Sorry I should have said it was p values from a t-test. – Stuart Brown May 27 '16 at 23:18

You could build a permutation test. The basic idea is find the null distribution of the minimum p-values (equivalently, the maximum statistic of the test) when you do 400 tests, one for variable. This is a multivariate permutation approach, taking into account the correlation structure of your data

First, calculate the unadjusted p-value with the observed response (you already have this)

The process of permutations is the next:

1. Permute the response variable (the 2 conditions in your case)
2. Do the 400 tests with the permuted response
3. Save the minimun p-value of the tests from these 400 p-values
4. Repeat the process 1000 or 10000 times

Finally, you calculate the adjusted p-value. For each unadjusted p-value, the adjusted p-value is the number of p-values obtained from the permutations smaller than that p-value, divided by the number of permutations.

• Thank you for that, seems like that procedure is used in the FDR-AME package mentioned by Jacky1. – Stuart Brown May 27 '16 at 23:46

BY is known to be a more stringent procedure than BH, thus it will select less discoveries than BH. So it is not surprizing that the adjusted pvalues are higher with BY than with BH: for a same threshold, you will reject more null hypotheses.

To deal with dependencies, there is a R package named FAMT for large-scale significance testing under dependence. You can also go to see the R package FDR-AME where there is a resampling-based multiple testing procedures that you can use in presence of high-correlation. Note that the resampling-based procedures (using permutations) will be more powerful if you get a high number of measures for each variable in each condition. For instance, if you get just 3 measures in 2 conditions, you will get a small number of possible different permutations, what will limit the procedure.

• Thanks for that I didn't realise that BY was more stringent. Thanks for your suggestions, I'll try them out, seems to be what I am looking for. – Stuart Brown May 27 '16 at 23:30