Marginalizing statistical test results in a two-factor grid

Question

Consider the following toy problem. Suppose that we have several food groups: meat, vegetables, fruit, etc. For any given pair of food types A and B, we ask whether people that eat both A AND B have a different outcome (e.g., live longer) than people who eat A OR B, but not both. After traversing all possible pairs of food categories and applying our test to each, we arrive at a set of p-values:

$Test( meat, vegetables ) \rightarrow p_{12} \\ Test( meat, fruit ) \rightarrow p_{13} \\ ...\\ Test( food_i, food_j ) \rightarrow p_{ij} $

The question is whether $p_{ij}$ values can now be used to determine whether any particular food group consistently leads to a low p value. The Fisher's method appears to be a good first pass at combining $p_{ij}$ for any fixed $i$, but I suspect that it doesn't properly account for the correlation structure in the data. In particular, there is likely a non-zero intersection between "people who eat meat AND vegetables" and "people who eat meat AND fruit", implying that the corresponding $Test(meat,vegetables)$ and $Test(meat,fruit)$ may not be independent.

What is the proper statistical procedure to summarize our grid of $p_{ij}$ values into a meaningful statistic for each $food_i$, assuming that we also have access to the correlation measure for any two tests?

If it makes a difference:

• In our actual application, the Test is Wilcoxon Rank Sum
• $p_{ii}$ is not well-defined. Food groups are NOT tested against themselves. For example, $Test(meat, meat)$ is not meaningful.

Answer 1

The solution that ended up working well for us was the recently-proposed harmonic mean p-value. Following the example in the original question, let's assume that there are $n$ food groups. For group $i$, we combine its corresponding p values according to:

$p_i = \frac{n-1}{\sum_{j \neq i} \frac{1}{p_{ij}}}$

As discussed in the paper, this combination is

i) Robust to positive dependency between p-values.

ii) Insensitive to the exact number of tests.

iii) Robust to the distribution of weights w.

iv) Most influenced by the smallest p-values.

where we kept the weights $w$ fixed at 1 for all tests. (The weights can be interpreted as relative prior belief about certain alternative hypotheses being more likely than others.)