6
$\begingroup$

A common approach for controlling the FWER in genomics analysis is the so-called minP/maxT procedure. See, for example:

Dudoit, S., Shaffer, J.P., and Boldrick, J.C. (2003). Multiple Hypothesis Testing in Microarray Experiments. Statistical Science, 18(1), 71–103.

I am having some trouble understanding the procedure in terms of its assumptions and theoretical properties. My understanding of the procedure is that it is conducted as follows:

The primary goal of the procedure is to construct an empirical null distribution of the test statistic against which to compare the observed values of the test statistic for each comparison in the study. You permute the data some arbitrarily large number of times (let's say 1,000 for simplicity). For each permutation, you calculate K test statistics (let's say K=10 for this example just to keep things simple). You choose the most extreme of these 10 test statistics (with 'most extreme' depending on the exact test and hypothesis in question, but in general this will correspond to the largest absolute value) for each permutation (for maxT; you can also select the lowest p-value, which would be minP - these are equivalent approaches). So, in the end, you would have an empirical distribution made up of 1,000 values, each of which is the largest of the 10 test statistics calculated in each of those permutations. You compare the observed values of those test statistics against this empirical distribution to construct your "final" p-value.

Please let me know if I have misrepresented or misunderstood the basic process of carrying out the procedure in any way.

The aspect of the procedure I do not understand is why we are selecting the largest/most extreme test statistic in each permutation to construct this empirical distribution. Since we typically interpret larger values of test statistics as being a measure of increasing evidence against the null hypothesis, isn't this empirical distribution one that is not necessarily representative of what the "true" distribution of test statistics should be under the null hypothesis?

For example, if we are just using two-sample t-tests for each comparison, under the "actual" null hypothesis, the distribution of t statistics should be centered around 0 with some positive variance. However, it seems that when using the minP/maxT procedure, we are comparing the observed t statistics against a distribution that is centered around the mean of the maximum T statistics, rather than 0. So, it seems to me that this procedure is controlling for the FWER by shifting the null hypothesis itself.

Is this interpretation correct? Have I missed some glaring fact that assuages my concerns?

If this is correct, what are the implications for inference? It seems to me that the nominal null hypothesis and the actual null hypothesis differ, which may lead to misleading interpretations of results.

EDIT:

It is possible there is an answer here in extreme value theory. The distribution of the maximum T statistic will asymptotically converge to a type II extreme value distribution (also known as a Fréchet distribution). Running a few quick simulations confirms this theoretical result.

However, it is not clear to me that this dismisses the concerns I have raised.

$\endgroup$
2
  • $\begingroup$ A similar question was asked here, but received no answer: stats.stackexchange.com/questions/97820/… $\endgroup$ Aug 11, 2015 at 18:35
  • $\begingroup$ Every source/comment I can find on the subject simply notes that using the maximum controls for multiple comparisons. While it certainly DOES control for multiple comparisons, my concern is that it is also leading to misleading inference by changing the null comparison. $\endgroup$ Aug 11, 2015 at 18:36

1 Answer 1

2
$\begingroup$

if you feel comfortable with the Bonferroni correction, you should not have a problem with minP method.

Suppose that you have K p-values: $P_1,P_2,...,P_K$. Let's consider the Bonferroni correction, each p-value is adjusted by the number of tests, $P_i^{Bon}= K*P_i$, then you compare your p-values with $\alpha$ to determine the significance.

Pretty simple, correct? But what is the null distribution of your p-values after the correction? Clearly, they are not uniform(0,1) anymore, right? Actually, they are unform(0,K) distributed. It is pretty similar to your question, the p-values are truely not uniform(0,1) distributed, but we compare them with uniform(0,1) to determine the significance.

The minP/maxT method adopts pretty much the same idea. We compare the statistics with a distribution which they are truly not from. This does not imply your null hypothesis has been changed. We have to correct our statistic to avoid any false positive and control the FWER. Therefore, we compare Bonferroni adjusted p-values with uniform(0,1) but they are truly from U(0,K). Likewise, we compare the t-test statistics in your example with the maximum T statistic where they are truly from a t distribution.

If you want some math proof, I posted it on another question that you referred: Why does max-t methods use only the maximum of generated t-values?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.