I'm implementing a permutation test for a test statistic with unknown distribution. Apart from my concrete situation (see below), I'd like to know first
- If and under what circumstances permutation tests might inflate the type I error, that is, in what kind of situations they may produce anti-conservative results?
In my concrete example, I'm trying a modification of the 'optimal cutpoint'-approach (see for instance "Practical p-value adjustment for optimally selected cutpoint" by S. Hilsenbeck and G. Clark from 1996 in Stat. Med. (15): 103-112). But instead of dichotomizing the whole continuous scale, I start with a fixed group at the lower end of the continuous scale and compare this group with a group from the higher end of the continuous scale (the data in the middle are ignored). In a systematic fashion, only the upper cutpoint is moved towards lower values of the continuous predictor searching for a maximized effect. In order to adjust for the maximum p-value selected from this search, the data is resampled in a permutation approach. However, in a simulation study with 1000 repetitions with each 500 permutations, the type I error seems to be inflated (i.e., under a postulated null effect, the proportion of significant test results exceeds the nominal alpha niveau). So my second, more concrete question is
- Are the simulation results reasonable and what might be an explanation for the results? Are 500 permutations too few and not precise enough (however, the deviances from the nominal alpha niveau are almost exclusively above the nominal alpha niveau)? What could be done to fix this problem?
I'd be glad for any kind of help!