Permutation tests and multiple testing

Question

I'm using permutation tests, based on random forest importance measures, to perform variables selection in my dataset. Using $99$ permutations, my permutation tests $p$-values are comprised between $0.01$ and $1$.

As I am using this procedure with more than $300$ covariates, I considered the issue of multiple testing. However, with all the $p$-value adjustment procedures I considered (BH, q-value, etc.) none of my $p$-values remained significant. Is there an easy way to deal with multiple testing when using permutation tests?

Answer 1

You'll have to either (a) increase the number of permutations so the p-values won't have a lower bound as high as 0.01 (even with a single measure I'd use at least 1000 permutations!) or (b) incorporate multiple testing correction within the permutation testing. This can be achieved by the Max-T method: in each permutation, record the maximal statistic across all the 300 measures. You'll end up with a single empirical sampling distribution that describes the maximal statistic across all measures, given the null hypothesis. Then, for each measure in the unpermuted set, determine its p-value according to this distribution. The resulting 300 p-values will be already corrected for family-wise error.

Reference for the max-t/min-p method:

Westfall, P. H., & Young, S. S. (1993). Resampling-based multiple testing: Examples and methods for p-value adjustment (Vol. 279). John Wiley & Sons.