Multiple comparisons - adjusting p values when some variables are correlated I'm working with 400+ variables (mostly 2D gel data) across 2 conditions for which I've calculated p values with a students t-test. Some of the variables are highly correlated. I'm wanting to adjust the p values to allow for multiple comparisons and I've used p.adjust in R with the "BH" method but I understand that it supposed to be for independent variables. I also tried the "BY" method as I thought being "under dependency" meant that it allowed for some variables to be non-independent (ie correlated), and so I assumed the adjusted p values would be lower than with the "BH" method, but they're not. (When I use the p.adjust BY method on my data the adjusted p values are all 1). Is there a method of adjusting p values that allows for some correlated variables? Thanks for any help.
> pvals$adjusted_BY <- p.adjust(pvals$p_value, method = "BY")
> pvals$adjusted_BH <- p.adjust(pvals$p_value, method = "BH")
> head(pvals)
    ID     p_value adjusted_BY adjusted_BH
1 2366 0.001361787           1   0.5828448
2  746 0.008377390           1   0.9516818
3 1635 0.017267163           1   0.9516818
4 1577 0.025868990           1   0.9516818
5 1588 0.029862316           1   0.9516818
6  891 0.036119770           1   0.9516818

 A: You could build a permutation test. The basic idea is find the null distribution of the minimum p-values (equivalently, the maximum statistic of the test) when you do 400 tests, one for variable. This is a multivariate permutation approach, taking into account the correlation structure of your data
First, calculate the unadjusted p-value with the observed response (you already have this)
The process of permutations is the next:


*

*Permute the response variable (the 2 conditions in your case)

*Do the 400 tests with the permuted response

*Save the minimun p-value of the tests from these 400 p-values

*Repeat the process 1000 or 10000 times


Finally, you calculate the adjusted p-value. For each unadjusted p-value, the adjusted p-value is the number of p-values obtained from the permutations smaller than that p-value, divided by the number of permutations.
A: BY is known to be a more stringent procedure than BH, thus it will select less discoveries than BH. So it is not surprizing that the adjusted pvalues are higher with BY than with BH: for a same threshold, you will reject more null hypotheses.
To deal with dependencies, there is a R package named FAMT for large-scale significance testing under dependence. You can also go to see the R package FDR-AME where there is a resampling-based multiple testing procedures that you can use in presence of high-correlation. Note that the resampling-based procedures (using permutations) will be more powerful if you get a high number of measures for each variable in each condition. For instance, if you get just 3 measures in 2 conditions, you will get a small number of possible different permutations, what will limit the procedure.
