Combining p-values from different statistical tests applied on the same data

Question

Although the title of the question appears trivial, I would like to explain that it is not that trivial in the sense that it's different from the question of applying the same statistical test in similar datasets to test against a total null hypothesis (meta-analysis, e.g. using Fisher's method for combining p-values). What I am looking for, is a method (if it exists and if the question is valid in statistical terms) that would combine p-values from two different statistical tests (e.g. a t-test and a u-test, even if one is parametric and the other is not), applied to compare the centers of two samplings from two populations. So far I have searched a lot on the web without a clear answer. The best answer I could find was based on game theory concepts by David Bickel (http://arxiv.org/pdf/1111.6174.pdf).

A very simplistic solution would be a voting scheme. Suppose that I have two vectors of observations $A=[a_1, a_2, ..., a_n]$ and $B=[b_1, b_2, ..., b_n]$ and I want to apply several t-like statistics (t-test, u-test, even 1-way ANOVA) to test the hypothesis that the centers (means, medians etc.) of the two undlerlying distributions are equal against the hypothesis that they are not, at a significance level of 0.05. Suppose that I run 5 tests. Would it be legitimate to say that there is sufficient evidence to reject the null distribution if I have a p-value < 0.05 in 3 out of 5 tests?

Would another solution be to use the law of total probability or this is completely wrong? For example, suppose that $A$ is the event that the null distribution is rejected. Then, using 3 tests, $T_1$, $T_2$, $T_3$ (meaning that $P(T_1)=P(T_2)=P(T_3)=1/3$), would a possible value for $P(A)$ be $P(A) = P(A|T_1)P(T_1) + P(A|T_2)P(T_2) + P(A|T_3)P(T_3)$, where $P(A|T_i)$ is the probability that the null distribution is rejected under the test $T_i$.

I apologize if the answer is obvious or the question too stupid

Answer 1

Using multiple testing correction as advocated by Corone is ok, but it will cost you mountains of power as your p-values will generally be well correlated, even using Hommel correction.

There is a solution which is computation demanding but will do much better in term of power. If $p_1, p_2, \dots, p_n$ are your p-values, let $p^* = \min (p_1, \dots, p_n)$. Consider that $p^*$ is your new test statistic: the smaller it is, the stronger it advocates against the null hypothesis.

You need to compute $p$-value for the observed value of $p^*$ (call it $p^*_{obs}$). For this, you can simulate, say, 100 000 data sets under the null hypotheses, and for each such data set, compute a $p^*$. This gives you an empirical distribution of $p^*$ under the null hypothesis. Your $p$-value is the proportion of simulated values which are $<p^*_{obs}$.

How do you simulate the data sets under the null hypothesis? In your case you have, if I guess well, cases and controls, and RNS-seq data to estimate expression levels. To simulate a data set under the null, it is customary to simply randomly permute the case/control status.

Answer 2

This sort of thing would usually covered by multiple hypothesis testing, although it isn't quite a typical situation.

You are correct in noting that this is different from meta-analysis, in that you are using the same data for multiple tests, but that situation is still covered by multiple-hypothesis testing. What is slightly odd here is that it is almost the same hypothesis that you are testing multiple times, and then you want the global null hypothesis that is the intersection of all of those - it is perhaps worth wondering why you feel the need to do this, but there could be legitimate reasons.

Were you doing a more analytically tractable set of tests, one might head down the Union-Intersection test route, but I don't think that would get you anywhere, so I'd recommend using an out of the box multiplicity correction.

I'd suggest you start by having a look at what Wikipedia has to say on the subject, but try not to get too bogged down: http://en.wikipedia.org/wiki/Multiple_comparisons

So, you need to use a multiplicity correction, and ruling out Union-Intersection, roughly your options are as follows

• Bonferonni - Strictly dominated by Holm-Bonferroni, historical interest only
• Holm-Bonferroni - Will work for you, but will cost you power (possibly a lot in your case)
• Sidak - more powerful than BH, but you cannot use this because your p-values will be correlated
• Hommel - more powerful than BH, and you should be fine, since your p-values are undoubtably positively correlated

Your biggest issue is that you are very likely to get very similar p-values in your different tests. Hommel shouldn't punish you too much for this.

For example, you can adjust p-values in R using p.adjust

p = c(0.03, 0.034, 0.041)
p.adjust(p, method = "bonferroni")
p.adjust(p, method = "holm")
p.adjust(p, method = "hommel")

> p.adjust(p, method = "bonferroni")
[1] 0.090 0.102 0.123
> p.adjust(p, method = "holm")
[1] 0.09 0.09 0.09
> p.adjust(p, method = "hommel")
[1] 0.041 0.041 0.041

These methods all control the Family-wise Error Rate which means that if you test each p-value in turn based on it passing your threshold, then the probability of 1 or more errors is still controlled at $\alpha$. This means that you can reject the global hypothesis if you reject one or more sub-hypothesis, and the size of your test is still controlled at $\alpha$.

As I intimated at the start, this won't be the most powerful attack you could do, but anything more sophisticated is going to require much more work.

---

Why this controls $\alpha$

The global null hypothesis is that all child null hypothesis are true.

Let the outcome of a single test be $X_i$ taking the value 1 if the null is rejected, 0 otherwise.

Since $X_i$ are undoubtedly positively correlated, we can use Hommel to control for the FWER.

This control means that the probability that one or more tests falsely reject is controlled at $\alpha$

Therefore, $P(\sum(X_i) > 0) \le \alpha$

Therefore if you reject the global hypothesis if one or more child hypotheses are rejected, the size of the global test is $\le \alpha$

Answer 3

In case you are still interested, I found this paper that seems to address the problem of integrating different analysis algorithms on the same set of RNA-seq data: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4344485/. It's basically proposing a weighted sum of all p-values and the weights are based on the ROC of each algorithm. The name of the paper is: Systematic integration of RNA-Seq statistical algorithms for accurate detection of differential gene expression patterns.