# Controlling type I error in permutation tests

I'm implementing a permutation test for a test statistic with unknown distribution. Apart from my concrete situation (see below), I'd like to know first

1. If and under what circumstances permutation tests might inflate the type I error, that is, in what kind of situations they may produce anti-conservative results?

In my concrete example, I'm trying a modification of the 'optimal cutpoint'-approach (see for instance "Practical p-value adjustment for optimally selected cutpoint" by S. Hilsenbeck and G. Clark from 1996 in Stat. Med. (15): 103-112). But instead of dichotomizing the whole continuous scale, I start with a fixed group at the lower end of the continuous scale and compare this group with a group from the higher end of the continuous scale (the data in the middle are ignored). In a systematic fashion, only the upper cutpoint is moved towards lower values of the continuous predictor searching for a maximized effect. In order to adjust for the maximum p-value selected from this search, the data is resampled in a permutation approach. However, in a simulation study with 1000 repetitions with each 500 permutations, the type I error seems to be inflated (i.e., under a postulated null effect, the proportion of significant test results exceeds the nominal alpha niveau). So my second, more concrete question is

1. Are the simulation results reasonable and what might be an explanation for the results? Are 500 permutations too few and not precise enough (however, the deviances from the nominal alpha niveau are almost exclusively above the nominal alpha niveau)? What could be done to fix this problem?

I'd be glad for any kind of help!

-

This happens all the time when the choice of the rejection region is such that its probability under the NULL of being in that region not being less than or equal to say .05.

It is usually considered invalid to choose a rejection region that occurs with probability greater than the target type one error rate – a valid test must have a smaller rejection region. So choose a smaller rejection region. With permutation tests, that means you have to do a better job of pinning down the distribution of the test statistic you are obtaining by simulation?

Recall that for independent draws of binary outcomes (reject, don’t reject) the variance is p * (1 – p) and the proportion estimated from n samples p * (1 - p)/n so make that small enough by drawing n samples.

-