Combining p-values from different statistical tests applied on the same data

Although the title of the question appears trivial, I would like to explain that it is not that trivial in the sense that it's different from the question of applying the same statistical test in similar datasets to test against a total null hypothesis (meta-analysis, e.g. using Fisher's method for combining p-values). What I am looking for, is a method (if it exists and if the question is valid in statistical terms) that would combine p-values from two different statistical tests (e.g. a t-test and a u-test, even if one is parametric and the other is not), applied to compare the centers of two samplings from two populations. So far I have searched a lot on the web without a clear answer. The best answer I could find was based on game theory concepts by David Bickel (http://arxiv.org/pdf/1111.6174.pdf).

A very simplistic solution would be a voting scheme. Suppose that I have two vectors of observations $A=[a_1, a_2, ..., a_n]$ and $B=[b_1, b_2, ..., b_n]$ and I want to apply several t-like statistics (t-test, u-test, even 1-way ANOVA) to test the hypothesis that the centers (means, medians etc.) of the two undlerlying distributions are equal against the hypothesis that they are not, at a significance level of 0.05. Suppose that I run 5 tests. Would it be legitimate to say that there is sufficient evidence to reject the null distribution if I have a p-value < 0.05 in 3 out of 5 tests?

Would another solution be to use the law of total probability or this is completely wrong? For example, suppose that $A$ is the event that the null distribution is rejected. Then, using 3 tests, $T_1$, $T_2$, $T_3$ (meaning that $P(T_1)=P(T_2)=P(T_3)=1/3$), would a possible value for $P(A)$ be $P(A) = P(A|T_1)P(T_1) + P(A|T_2)P(T_2) + P(A|T_3)P(T_3)$, where $P(A|T_i)$ is the probability that the null distribution is rejected under the test $T_i$.

I apologize if the answer is obvious or the question too stupid

• What does $P(T_i)$ represent in that law of total probability calculation? – Glen_b Nov 15 '13 at 10:51
• I am sorry that I cannot give a mathematical explanation of what you are looking for, but I happen to know that a software we develop in our lab has this feature implemented: check here on how to do it: gitools.org/documentation/UserGuide_Combinations.html and here the implementation: github.com/gitools/gitools/blob/…. I will check back when I find the formula in the original paper. – dmeu Nov 15 '13 at 10:52
• @Glen_b P(Ti) represents the "probability" of using the statistical test Ti. I know that this is not exactly a probability in a strict manner. It's rather a weight that says that I have used n tests for the same dataset. – Panos Nov 15 '13 at 10:56
• @dmeu Thanks! However I think that your software answers the trivial part (see above, combination of multiple datasets using a single test) and not my question. Sorry if this is not the case. – Panos Nov 15 '13 at 11:04
• @Panos you might be right. I read different as "two independant tests (different) of the same kind". Sorry to get your hopes up. – dmeu Nov 15 '13 at 11:07

Using multiple testing correction as advocated by Corone is ok, but it will cost you mountains of power as your p-values will generally be well correlated, even using Hommel correction.

There is a solution which is computation demanding but will do much better in term of power. If $p_1, p_2, \dots, p_n$ are your p-values, let $p^* = \min (p_1, \dots, p_n)$. Consider that $p^*$ is your new test statistic: the smaller it is, the stronger it advocates against the null hypothesis.

You need to compute $p$-value for the observed value of $p^*$ (call it $p^*_{obs}$). For this, you can simulate, say, 100 000 data sets under the null hypotheses, and for each such data set, compute a $p^*$. This gives you an empirical distribution of $p^*$ under the null hypothesis. Your $p$-value is the proportion of simulated values which are $<p^*_{obs}$.

How do you simulate the data sets under the null hypothesis? In your case you have, if I guess well, cases and controls, and RNS-seq data to estimate expression levels. To simulate a data set under the null, it is customary to simply randomly permute the case/control status.

• +1 Yes, this is one of the approaches I was meaning with "more work". However, it should be noted that it isn't a given that taking the smallest p-value is the best approach here. 99 p-values close to 0.5 and one at 0.02 is very different to 99 p-values close to 0.02. Once you open the door to resampling the null, then it would be worth looking at "voting" methods, since consistency between the tests may be as (more) important that fluking a low p-value on a single test. – Corone Nov 15 '13 at 12:37
• Yep, you’re right. However for most association tests I think taking the min is a good idea. With more work a ''maximin efficiency robust test'' could be constructed from the different test, but this really requires to work on the tests... – Elvis Nov 15 '13 at 13:06
• yep, not to mention thing will get really hairy/exciting/fun if we start to worry about the fact that some tests will be more powerful that other test - in an ideal world you want to listen most to the most powerful tests... – Corone Nov 15 '13 at 13:11
• @Elvis Something like this is the closest alternative to something involving the law of total probability (which proved wrong) that I had in mind. I was trying to think of a resampling procedure but you formalized it perfectly! Computational power will not be an issue at this point (fortunately!). As for voting, one could incorporate something like the Whitlock's method for meta-analysis (ncbi.nlm.nih.gov/pmc/articles/PMC3135688) but giving weight to the statistical test. Such info could be derived from relative publications (e.g. biomedcentral.com/1471-2105/14/91). – Panos Nov 15 '13 at 13:56
• yep! you can take any function $p^* = f(p_1, \dots, p_n)$ of the $p$-values, as long as it is non-decreasing with respect to each $p_i$. – Elvis Nov 15 '13 at 16:27

This sort of thing would usually covered by multiple hypothesis testing, although it isn't quite a typical situation.

You are correct in noting that this is different from meta-analysis, in that you are using the same data for multiple tests, but that situation is still covered by multiple-hypothesis testing. What is slightly odd here is that it is almost the same hypothesis that you are testing multiple times, and then you want the global null hypothesis that is the intersection of all of those - it is perhaps worth wondering why you feel the need to do this, but there could be legitimate reasons.

Were you doing a more analytically tractable set of tests, one might head down the Union-Intersection test route, but I don't think that would get you anywhere, so I'd recommend using an out of the box multiplicity correction.

I'd suggest you start by having a look at what Wikipedia has to say on the subject, but try not to get too bogged down: http://en.wikipedia.org/wiki/Multiple_comparisons

So, you need to use a multiplicity correction, and ruling out Union-Intersection, roughly your options are as follows

• Bonferonni - Strictly dominated by Holm-Bonferroni, historical interest only
• Holm-Bonferroni - Will work for you, but will cost you power (possibly a lot in your case)
• Sidak - more powerful than BH, but you cannot use this because your p-values will be correlated
• Hommel - more powerful than BH, and you should be fine, since your p-values are undoubtably positively correlated

Your biggest issue is that you are very likely to get very similar p-values in your different tests. Hommel shouldn't punish you too much for this.

For example, you can adjust p-values in R using p.adjust

p = c(0.03, 0.034, 0.041)

[1] 0.090 0.102 0.123
[1] 0.09 0.09 0.09
[1] 0.041 0.041 0.041


These methods all control the Family-wise Error Rate which means that if you test each p-value in turn based on it passing your threshold, then the probability of 1 or more errors is still controlled at $\alpha$. This means that you can reject the global hypothesis if you reject one or more sub-hypothesis, and the size of your test is still controlled at $\alpha$.

As I intimated at the start, this won't be the most powerful attack you could do, but anything more sophisticated is going to require much more work.

Why this controls $\alpha$

The global null hypothesis is that all child null hypothesis are true.

Let the outcome of a single test be $X_i$ taking the value 1 if the null is rejected, 0 otherwise.

Since $X_i$ are undoubtedly positively correlated, we can use Hommel to control for the FWER.

This control means that the probability that one or more tests falsely reject is controlled at $\alpha$

Therefore, $P(\sum(X_i) > 0) \le \alpha$

Therefore if you reject the global hypothesis if one or more child hypotheses are rejected, the size of the global test is $\le \alpha$

• Thank you for your quick reply! The main idea is to combine several statistical algorithms that detect differential gene expression using RNA-Seq data, with a more sophisticated way than multiplying the p-values which is not correct anyway. What you suggest is similar to what I found in researchgate.net/publication/… However, I wonder if the voting scheme or the law of total probability has any meaning in this context. – Panos Nov 15 '13 at 11:01
• @Panos in that case, then yes I'd definitely suggest something like this. The downside is that the control is conservative, because we are controlling P(1 or More Selected are False) instead of P(All Selected are False), but since your algorithms are likely to be quite positively correlated, the difference likely won't be huge. – Corone Nov 15 '13 at 11:03
• Not to mention, one shouldn't get too wedded to a particular size threshold either - as is often said there is nothing particularly special about 0.05 or 0.01. – Corone Nov 15 '13 at 11:05
• @Panos if you were to head down the direction of working out the correct probabilities for your voting scheme, then you would ultimately re-derive the various multiplicity test I suggested, the exact one depending on what assumptions you make along the way. – Corone Nov 15 '13 at 11:12
• @Benjamin while I'd also advocate reading about it (it is important!), I don't think it would be at all appropriate here. FDR is controlling the expected proportion of false positives from a number of selections. Since all Panos' tests are trying to inform him on the same global hypothesis, it would make little sense to control the expected proportion of false test - especially when you have correlation. He isn't choosing between multiple different questions, but trying many ways of answering the same question. – Corone Nov 15 '13 at 12:43