I'm working with 400+ variables (mostly 2D gel data) across 2 conditions for which I've calculated p values with a students t-test. Some of the variables are highly correlated. I'm wanting to adjust the p values to allow for multiple comparisons and I've used p.adjust in R with the "BH" method but I understand that it supposed to be for independent variables. I also tried the "BY" method as I thought being "under dependency" meant that it allowed for some variables to be non-independent (ie correlated), and so I assumed the adjusted p values would be lower than with the "BH" method, but they're not. (When I use the p.adjust BY method on my data the adjusted p values are all 1). Is there a method of adjusting p values that allows for some correlated variables? Thanks for any help.

> pvals$adjusted_BY <- p.adjust(pvals$p_value, method = "BY")
> pvals$adjusted_BH <- p.adjust(pvals$p_value, method = "BH")
> head(pvals)
    ID     p_value adjusted_BY adjusted_BH
1 2366 0.001361787           1   0.5828448
2  746 0.008377390           1   0.9516818
3 1635 0.017267163           1   0.9516818
4 1577 0.025868990           1   0.9516818
5 1588 0.029862316           1   0.9516818
6  891 0.036119770           1   0.9516818
  • $\begingroup$ Could you provide a minimal code example? That might help get a better answer. $\endgroup$ – Fabian Fagerholm May 27 '16 at 6:18
  • $\begingroup$ "adjust the p values" How did you derive the p-values? What kind of test did you conduct? $\endgroup$ – Roland May 27 '16 at 7:08
  • $\begingroup$ Sorry I should have said it was p values from a t-test. $\endgroup$ – Stuart Brown May 27 '16 at 23:18

You could build a permutation test. The basic idea is find the null distribution of the minimum p-values (equivalently, the maximum statistic of the test) when you do 400 tests, one for variable. This is a multivariate permutation approach, taking into account the correlation structure of your data

First, calculate the unadjusted p-value with the observed response (you already have this)

The process of permutations is the next:

  1. Permute the response variable (the 2 conditions in your case)
  2. Do the 400 tests with the permuted response
  3. Save the minimun p-value of the tests from these 400 p-values
  4. Repeat the process 1000 or 10000 times

Finally, you calculate the adjusted p-value. For each unadjusted p-value, the adjusted p-value is the number of p-values obtained from the permutations smaller than that p-value, divided by the number of permutations.

  • $\begingroup$ Thank you for that, seems like that procedure is used in the FDR-AME package mentioned by Jacky1. $\endgroup$ – Stuart Brown May 27 '16 at 23:46

BY is known to be a more stringent procedure than BH, thus it will select less discoveries than BH. So it is not surprizing that the adjusted pvalues are higher with BY than with BH: for a same threshold, you will reject more null hypotheses.

To deal with dependencies, there is a R package named FAMT for large-scale significance testing under dependence. You can also go to see the R package FDR-AME where there is a resampling-based multiple testing procedures that you can use in presence of high-correlation. Note that the resampling-based procedures (using permutations) will be more powerful if you get a high number of measures for each variable in each condition. For instance, if you get just 3 measures in 2 conditions, you will get a small number of possible different permutations, what will limit the procedure.

  • $\begingroup$ Thanks for that I didn't realise that BY was more stringent. Thanks for your suggestions, I'll try them out, seems to be what I am looking for. $\endgroup$ – Stuart Brown May 27 '16 at 23:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.