# How to apply FDR when you have two types of test statistics?

I'm testing the bivariate correlations among a large dataset of plant traits. Some of these correlations have highly non-normal residuals that are normal when log-transformed, but other correlations are the other way around- they have normal residuals when untransformed that become very non-normal when logged. I didn't want to log traits for some correlations but not others, so to deal with this problem, I decided to test a spearman correlation and a rank correlation for each pair of traits, and only say that a correlation between these traits is strongly supported if both correlations are significant.

Because I have a large number of trait combinations (~250), and positive dependence among the test statistics, I want to use the Benjamini-Hochberg FDR method to adjust the p-values. But, I'm not sure how to account for the fact I'm doing two kinds of tests. Should I:

1) Conduct one B-H adjustment on the rank correlations, assuming 250 tests, and then a separate B-H adjustment on the spearman correlations, also assuming 250 tests?

2) Pool the p-values for both kinds of correlations and conduct one B-H adjustment assuming 500 tests?

or 3) Get rid of the comparison altogether and only test rank correlations?

Thanks so much for any insight!

Short version: Stick with 3).

Longer version: First of all, you talk about "rank" and "Spearman" correlations. I usually interpret "rank correlations" to be precisely "Spearman" correlations, so maybe you should clarify that. In any case, what I am writing below holds if both test statistics correspond to the same null hypothesis.

If you apply 1) and 2) at level $\alpha$ and then reject hypotheses for which both tests have been rejected, then the actual significance level will be smaller than $\alpha$.

A legit method would be the following: For each hypothesis, take the smallest of the two p-values and multiply it by $2$, i.e. if you have p-values $P_1,P_2$, then use $\tilde{P} = 2\min \{P_1,P_2\}$ as your new "p-value". Then apply Benjamini-Hochberg to these. Under the null $P_1, P_2$ are (sub)uniform and then $\tilde{P}$ will be subuniform, thus showing the validity of BH. (This is really just an application of Bonferroni's global test).

But note that even the above approach essentially penalizes you with roughly speaking a multiplicity factor of 2. I think that in your case this will almost always provide worse results than just using one robust statistic which applies to all of your dataset.

If your two statistics were less dependent themselves, then one could think of more refined ways of approaching this, so as to actually increase power, but again, this does not seem to apply to your case.

• Thanks for the explanation! I did mean "rank" and "Pearson", not "rank" and "Spearman", sorry. – quercus Apr 1 '16 at 23:01
• @quercus ok; just go with spearman then I'd say. – air Apr 1 '16 at 23:43
• +1 Nice answer. @quercusm you might consider trying varying values of $\lambda$ in the Box Cox transformation to potentially find one transformation which would fit the majority of your data. You could write a function in R or python to loop through allt he possible values and the number of comparisons which are normal under that value, then if you can fit a majority of your data set using that value you might be able to lump them together. Also Air, I hope your applications went well! Good lucK! – Chris C Apr 2 '16 at 14:31
• @ChrisC Indeed, the Box Cox transformation could be a more direct way to tackle the problem! Also thanks a lot :D, I did get into my dream school (the one with the nice weather :P)! – air Apr 3 '16 at 11:43