5
$\begingroup$

What is this about?

Suppose I have performed two statistical tests (with a continuous distribution of $p$ values) for the same one-sided research hypothesis on different datasets, yielding the $p$ values $p_1=x$ and $p_2=1-x$. I have no further a priori knowledge that makes a distinction between the datasets, e.g., to weigh them. In this case, I do not learn anything from my tests as to whether my research hypothesis is true or not. If I tested the opposite research hypothesis, I would obtain the same results (just in different order). Therefore the accurate result for combining these $p$ values is $p_\text{comb} = \frac{1}{2}$.

However many methods for combining $p$ values do not treat $p$ values and their complements in a symmetrical fashion and produce implausible results. For example for $x=0.01$:

  • Fisher’s method: $p_\text{comb} = 0.06$
  • Pearson’s method: $p_\text{comb} = 0.94$
  • Tippett’s method: $p_\text{comb} = 0.02$
  • Simes’ method: $p_\text{comb} = 0.02$

By contrast, Stouffer’s method, Mudholkar’s and George’s method, as well as Edgington’s method are symmetric (as described above) and produce $p_\text{comb} = \frac{1}{2}$.

Obviously, this problem extends beyond the above simple example and can lead to many clear false positives in many cases. I could probably produce datasets where two opposing one-side research hypotheses are both highly significant.

My question

I consider it a serious flaw of a combining method if complementary $p$ values do not cancel each other. However, I fail to find this issue mentioned in the literature on combining $p$ values. To give just one example, the paper Choosing Between Methods of Combining $p$-values does not mention it as far as I can tell. In fact, the only mention I have found so far is here on this site.

So, what am I missing?

  • Is there literature on this (and I failed to find it)?
  • Is my argument somehow flawed and I am overestimating the importance of this?
  • Is this generally accepted, but just not documented?
Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ This is another way of looking at the well-known fact that a two-tailed test from a meta-analysis may be designed to furnish a low p-value when results from different studies significantly depart from the null in different directions. Take the simple case of testing for a difference in means: would you still cling to the null hypothesis of a common, zero say, effect when you have a dozen studies showing a large increase & a dozen showing a large decrease? $\endgroup$
    – Scortchi ♦
    Commented Mar 15, 2022 at 11:31
  • $\begingroup$ @Scortchi: Either I misunderstand you or these are not comparable. Here I look at one-sided tests for the same direction, i.e., deviations from the null in different directions are properly distinguished in the individual tests. In fact, something similar to your last example may have been underlying my example (just with one study each, not a dozen): You have one study showing a large decrease and one showing a large increase (of same significance). (Of course, if I had dozens of each, I would probably conclude that something is horribly wrong, but I still cannot say anything about my trend.) $\endgroup$
    – Wrzlprmft
    Commented Mar 15, 2022 at 12:02
  • $\begingroup$ It's the same kettle of fish. If you encompass study heterogeneity in your alternative hypothesis, then there's nothing counter-intuitive about observing low p-values for both one-tailed tests; if you rule it out as a possibility then you need to choose a test that isn't sensitive to it (though of course risking an additional assumption violation). $\endgroup$
    – Scortchi ♦
    Commented Mar 15, 2022 at 12:54
  • $\begingroup$ I see. However, this directly poses the question (at least to me) why anybody would work with such a hypothesis in the first place. Also I wonder whether one can formulate better null/alternative hypotheses that reflect a consistent trend (or absence thereof). $\endgroup$
    – Wrzlprmft
    Commented Mar 15, 2022 at 13:52
  • $\begingroup$ "If I tested the opposite research hypothesis, I would obtain the same results (just in different order)." I might not be understanding you, but I do not think this is strictly true. For example, if you failed to reject $\text{H}_0 \text{: } \mu \ge c $, is that because $\mu \ge c$, or because your data do not have enough power for that null? It is quite possible that you could either reject or fail to reject a different null hypothesis, $\text{H}_0\text{: }\mu \le c$, on the same data. $\endgroup$
    – Alexis
    Commented Mar 15, 2022 at 16:33

1 Answer 1

Reset to default
2
$\begingroup$

I am not sure when the methods were first systematically compared but Loughin in a paper entitled "A systematic comparison of methods for combining p-values from independent tests" available here compared some of them. He does refer to previous work which discussed the issue you raise. Some other methods are compared in my R package metap and the comparisons set out in one of the vignettes. (Apologies for the self-promotion.)

I am not an expert on modern genetics but I believe that there the interest is in which tests display the signal and so a method which does not cancel is quite acceptable and indeed even preferred.

As you suggest plotting the $p$-values should really be obligatory as then it becomes clear that there are extreme values in both directions which should lead the investigator to question their theory. I am not aware of substantial sources recommending this though.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.