Consider the following toy problem. Suppose that we have several food groups: meat, vegetables, fruit, etc. For any given pair of food types A and B, we ask whether people that eat both A AND B have a different outcome (e.g., live longer) than people who eat A OR B, but not both. After traversing all possible pairs of food categories and applying our test to each, we arrive at a set of p-values:

$Test( meat, vegetables ) \rightarrow p_{12} \\ Test( meat, fruit ) \rightarrow p_{13} \\ ...\\ Test( food_i, food_j ) \rightarrow p_{ij} $

The question is whether $p_{ij}$ values can now be used to determine whether any particular food group consistently leads to a low p value. The Fisher's method appears to be a good first pass at combining $p_{ij}$ for any fixed $i$, but I suspect that it doesn't properly account for the correlation structure in the data. In particular, there is likely a non-zero intersection between "people who eat meat AND vegetables" and "people who eat meat AND fruit", implying that the corresponding $Test(meat,vegetables)$ and $Test(meat,fruit)$ may not be independent.

What is the proper statistical procedure to summarize our grid of $p_{ij}$ values into a meaningful statistic for each $food_i$, assuming that we also have access to the correlation measure for any two tests?

If it makes a difference:

  • In our actual application, the Test is Wilcoxon Rank Sum
  • $p_{ii}$ is not well-defined. Food groups are NOT tested against themselves. For example, $Test(meat, meat)$ is not meaningful.
  • 1
    $\begingroup$ "Whether any particular food group consistently leads to a low p value" is not a testable statistical hypothesis, because it depends fundamentally on the test procedure and the sample size: as such, it reveals nothing definite about the population of interest. Could you therefore explain what your actual scientific question is? $\endgroup$ – whuber Jun 18 '19 at 20:52
  • $\begingroup$ @whuber: we are interested in ranking our "food groups" based on the observed p_ij values. Can we ask "do tests that involve meat lead to significantly lower p values than tests that involve fruit (taking into account that individual tests are not independent)"? $\endgroup$ – Artem Sokolov Jun 18 '19 at 21:02
  • $\begingroup$ Our actual application is polypharmacology. Our "food groups" are cell surface receptors; our "people" are compounds that bind to them. We are studying interaction effects of targeting multiple receptors, and we would like to know if certain receptors are involved in more interactions than others. $\endgroup$ – Artem Sokolov Jun 18 '19 at 21:04
  • 1
    $\begingroup$ A ranking based on p-values is arbitrary and meaningless, for the same reasons I gave earlier. Aren't you really interested in some kind of meaningful ranking with scientific validity? $\endgroup$ – whuber Jun 18 '19 at 21:04
  • $\begingroup$ @whuber: We are happy to move upstream to modifying the actual Wilcoxon Rank Sum Test rather than working with the resulting p values directly. The ranking we're interested in is the level of interaction each "food group" has with others, summarized on a per-"food group", rather than per-interaction basis. $\endgroup$ – Artem Sokolov Jun 18 '19 at 21:13

The solution that ended up working well for us was the recently-proposed harmonic mean p-value. Following the example in the original question, let's assume that there are $n$ food groups. For group $i$, we combine its corresponding p values according to:

$p_i = \frac{n-1}{\sum_{j \neq i} \frac{1}{p_{ij}}}$

As discussed in the paper, this combination is

i) Robust to positive dependency between p-values.

ii) Insensitive to the exact number of tests.

iii) Robust to the distribution of weights w.

iv) Most influenced by the smallest p-values.

where we kept the weights $w$ fixed at 1 for all tests. (The weights can be interpreted as relative prior belief about certain alternative hypotheses being more likely than others.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.