Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I want to find out if there is a calculation that can give me the minimum sample size to achieve a certain statistical power for a Permutation Test. I'm not assuming any parametric distribution for populations from where the samples come from.
I've been searching but haven't been able to find a satisfying answer. Statistical power seems like an overlooked aspect on most texts on permutation tests.
You partially answered your own question. Consider the reason you're often performing a permutation test. It's usually in circumstances where you have little faith in any particular parametric distribution or for some other reason want a non-parametric solution. In that case, how does one estimate power? You could be doing the permutation test in situations where the populations have multimodal distributions, uniform distributions, any combination for your conditions that don't match, etc. All of those might have different power functions.
Keep in mind that the way power is estimated requires an assumption of some world where you know effect size and distribution of the effect. Since you have no notion of the latter you can't estimate power a priori. The best you could do is estimate power for some particular distribution you might think is close, subset of likely distributions, or taking your data as the population. But the latter leaves you know way to get to power other than post hoc.
I would wager permutation tests inflate the false negative rate in a small sample. Here's an extreme example to illustrate, but this applies with less extreme examples too:
Therefore: at least 1/24 ( = .042) permutations will have r=1 so p(r=1) >= 0.42. This is far greater than the real p-value of r=1, which is practically 0. The worst part is, with a p-threshold of .01, this result is not significant. Which is crazy; r=1 is the most significant possible.
This is because when you shuffle among existing values, you are not getting a true null distribution. The existing values are shuffled in order but not value. The values themselves are quantized. The values contributed to significance. This can, and will, bias the "null" shuffled distribution to something more significant. This holds for r-values less extreme than 1, and should be true for models other than correlation.
