3
$\begingroup$

I'm using permutation tests, based on random forest importance measures, to perform variables selection in my dataset. Using $99$ permutations, my permutation tests $p$-values are comprised between $0.01$ and $1$.

As I am using this procedure with more than $300$ covariates, I considered the issue of multiple testing. However, with all the $p$-value adjustment procedures I considered (BH, q-value, etc.) none of my $p$-values remained significant. Is there an easy way to deal with multiple testing when using permutation tests?

$\endgroup$
6
$\begingroup$

You'll have to either (a) increase the number of permutations so the p-values won't have a lower bound as high as 0.01 (even with a single measure I'd use at least 1000 permutations!) or (b) incorporate multiple testing correction within the permutation testing. This can be achieved by the Max-T method: in each permutation, record the maximal statistic across all the 300 measures. You'll end up with a single empirical sampling distribution that describes the maximal statistic across all measures, given the null hypothesis. Then, for each measure in the unpermuted set, determine its p-value according to this distribution. The resulting 300 p-values will be already corrected for family-wise error.

Reference for the max-t/min-p method:

Westfall, P. H., & Young, S. S. (1993). Resampling-based multiple testing: Examples and methods for p-value adjustment (Vol. 279). John Wiley & Sons.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.